[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:14 dist-test-slave-7zlw krb5kdc[29622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:14 dist-test-slave-7zlw krb5kdc[29622](info): set up 2 sockets
Sep 26 07:19:14 dist-test-slave-7zlw krb5kdc[29622](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 07:19:15.269690 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:15 dist-test-slave-7zlw krb5kdc[29622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:15.405853 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:35525
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:15.523262 29635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:15.523561 29635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:15.523603 29635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:15.527920 29635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:15.528002 29635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:15.528023 29635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:15.528040 29635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:15.528062 29635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:15.533663 29635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:35525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:15.534817 29635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:15.535907 29635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:15.545293 29643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:15.545369 29635 server_base.cc:1034] running on GCE node
W20260926 07:19:15.545315 29641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:15.545311 29640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:15.546249 29635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:15.547492 29635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:15.548707 29635 hybrid_clock.cc:648] HybridClock initialized: now 1790407155548679 us; error 44 us; skew 500 ppm
Sep 26 07:19:15 dist-test-slave-7zlw krb5kdc[29622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:15.551764 29635 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:15.552680 29635 webserver.cc:460] Webserver started at http://127.28.235.254:35295/ using document root <none> and password file <none>
I20260926 07:19:15.553293 29635 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:15.553362 29635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:15.553534 29635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:15.555596 29635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "da806653becf410f825f545e2c50675d"
format_stamp: "Formatted at 2026-09-26 07:19:15 on dist-test-slave-7zlw"
server_key: "b2710d7fdc64aa5c79c5a2b8a0900a7a"
server_key_iv: "024cd92b6e2f717ce89caa29cec7b23e"
server_key_version: "encryptionkey@0"
I20260926 07:19:15.556113 29635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "da806653becf410f825f545e2c50675d"
format_stamp: "Formatted at 2026-09-26 07:19:15 on dist-test-slave-7zlw"
server_key: "b2710d7fdc64aa5c79c5a2b8a0900a7a"
server_key_iv: "024cd92b6e2f717ce89caa29cec7b23e"
server_key_version: "encryptionkey@0"
I20260926 07:19:15.559896 29635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:19:15.562762 29650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:15.564189 29635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:19:15.564347 29635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "da806653becf410f825f545e2c50675d"
format_stamp: "Formatted at 2026-09-26 07:19:15 on dist-test-slave-7zlw"
server_key: "b2710d7fdc64aa5c79c5a2b8a0900a7a"
server_key_iv: "024cd92b6e2f717ce89caa29cec7b23e"
server_key_version: "encryptionkey@0"
I20260926 07:19:15.564450 29635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:15.580186 29635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:15.583737 29635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:15.583952 29635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:15.591390 29635 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:35525
I20260926 07:19:15.591408 29702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:35525 every 8 connection(s)
I20260926 07:19:15.592401 29635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:15.593324 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 29635
I20260926 07:19:15.593490 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:15.594797 29615 external_mini_cluster.cc:1428] Setting key 985b2755f64e807653ef88928aba2050
I20260926 07:19:15.595641 29703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:15.602264 29703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d: Bootstrap starting.
I20260926 07:19:15.604599 29703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:15.605449 29703 log.cc:824] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:15.607610 29703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d: No bootstrap required, opened a new log
I20260926 07:19:15.611198 29703 raft_consensus.cc:348] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da806653becf410f825f545e2c50675d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35525 } }
I20260926 07:19:15.611460 29703 raft_consensus.cc:374] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:15.611589 29703 raft_consensus.cc:729] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da806653becf410f825f545e2c50675d, State: Initialized, Role: FOLLOWER
I20260926 07:19:15.612180 29703 consensus_queue.cc:260] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da806653becf410f825f545e2c50675d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35525 } }
I20260926 07:19:15.612380 29703 raft_consensus.cc:388] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:15.612459 29703 raft_consensus.cc:482] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:15.612588 29703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:15.614002 29703 raft_consensus.cc:504] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da806653becf410f825f545e2c50675d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35525 } }
I20260926 07:19:15.614375 29703 leader_election.cc:302] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da806653becf410f825f545e2c50675d; no voters:
Sep 26 07:19:15 dist-test-slave-7zlw krb5kdc[29622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:15.614727 29703 leader_election.cc:288] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:15.614857 29708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:15.615127 29708 raft_consensus.cc:686] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [term 1 LEADER]: Becoming Leader. State: Replica: da806653becf410f825f545e2c50675d, State: Running, Role: LEADER
I20260926 07:19:15.615497 29708 consensus_queue.cc:237] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da806653becf410f825f545e2c50675d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35525 } }
I20260926 07:19:15.616215 29703 sys_catalog.cc:564] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:15.617486 29710 sys_catalog.cc:455] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [sys.catalog]: SysCatalogTable state changed. Reason: New leader da806653becf410f825f545e2c50675d. Latest consensus state: current_term: 1 leader_uuid: "da806653becf410f825f545e2c50675d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da806653becf410f825f545e2c50675d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35525 } } }
I20260926 07:19:15.617630 29710 sys_catalog.cc:458] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:15.617718 29709 sys_catalog.cc:455] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da806653becf410f825f545e2c50675d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da806653becf410f825f545e2c50675d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35525 } } }
I20260926 07:19:15.617839 29709 sys_catalog.cc:458] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d [sys.catalog]: This master's current role is: LEADER
W20260926 07:19:15.621569 29723 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:19:15.621644 29723 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:19:15.621701 29717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:15.622627 29717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:15.622747 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:15.598207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.28.235.254:35525)
0926 07:19:15.598690 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:15.598707 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:15.599806 (+ 1099us) server_negotiation.cc:407] Connection header received
0926 07:19:15.603374 (+ 3568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:15.603395 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:15.603757 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:15.604131 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:15.606990 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:15.608177 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:15.609573 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:15.610116 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:15.610326 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:15.615936 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:15.615943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:15.615959 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:15.615995 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:15.618639 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:15.619407 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:15.619413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:15.619419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:15.619572 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:15.620053 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:15.620057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:15.620060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:15.620586 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:15.620794 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:15.621517 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 07:19:15.621819 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":178,"threads_started":1}
I20260926 07:19:15.628577 29717 catalog_manager.cc:1347] Generated new cluster ID: ee051d681020426e95b94dc86ecdcca4
I20260926 07:19:15.628649 29717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:15.670265 29717 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:15.671590 29717 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:15.683372 29717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da806653becf410f825f545e2c50675d: Generated new TSK 0
I20260926 07:19:15.684248 29717 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:19:15.733373 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35525
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:15.857795 29729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:15.858075 29729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:15.858160 29729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:15.862323 29729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:15.862389 29729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:15.862488 29729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:15.867916 29729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:15.869109 29729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:15.870141 29729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:19:15.880211 29729 server_base.cc:1034] running on GCE node
W20260926 07:19:15.880156 29735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:15.880156 29737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:15.880177 29734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:15.880862 29729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:15.881644 29729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:15.882812 29729 hybrid_clock.cc:648] HybridClock initialized: now 1790407155882795 us; error 28 us; skew 500 ppm
Sep 26 07:19:15 dist-test-slave-7zlw krb5kdc[29622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:15.886185 29729 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:19:15.887123 29729 webserver.cc:460] Webserver started at http://127.28.235.193:40645/ using document root <none> and password file <none>
I20260926 07:19:15.887715 29729 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:15.887779 29729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:15.887996 29729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:15.890174 29729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "cc02c3d6920542e3801d7fa404c17071"
format_stamp: "Formatted at 2026-09-26 07:19:15 on dist-test-slave-7zlw"
server_key: "b573fd664f231e835361e007300358a3"
server_key_iv: "ca9b3daad844100c9e96b4ff83b32989"
server_key_version: "encryptionkey@0"
I20260926 07:19:15.890753 29729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "cc02c3d6920542e3801d7fa404c17071"
format_stamp: "Formatted at 2026-09-26 07:19:15 on dist-test-slave-7zlw"
server_key: "b573fd664f231e835361e007300358a3"
server_key_iv: "ca9b3daad844100c9e96b4ff83b32989"
server_key_version: "encryptionkey@0"
I20260926 07:19:15.894605 29729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:15.897222 29744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:15.898495 29729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:19:15.898602 29729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "cc02c3d6920542e3801d7fa404c17071"
format_stamp: "Formatted at 2026-09-26 07:19:15 on dist-test-slave-7zlw"
server_key: "b573fd664f231e835361e007300358a3"
server_key_iv: "ca9b3daad844100c9e96b4ff83b32989"
server_key_version: "encryptionkey@0"
I20260926 07:19:15.898757 29729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:15.929921 29729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:15.933421 29729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:15.933632 29729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:15.934288 29729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:15.935148 29729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:15.935199 29729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:15.935276 29729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:15.935309 29729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:15.944432 29729 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:44345
I20260926 07:19:15.944459 29857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:44345 every 8 connection(s)
I20260926 07:19:15.945423 29729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:15.951104 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 29729
I20260926 07:19:15.951196 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:15.951507 29615 external_mini_cluster.cc:1428] Setting key 9f59d74c650934a9794bca2d1a297289
Sep 26 07:19:15 dist-test-slave-7zlw krb5kdc[29622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:15.960520 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:15.947539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:38003 (local address 127.28.235.254:35525)
0926 07:19:15.947673 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:15.947679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:15.948734 (+ 1055us) server_negotiation.cc:407] Connection header received
0926 07:19:15.949732 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:15.949735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:15.949814 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:15.949906 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:15.951120 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:15.951812 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:15.952991 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:15.953321 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:15.953440 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:15.956240 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:15.956243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:15.956245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:15.956271 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:15.957862 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:15.958690 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:15.958694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:15.958695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:15.958792 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:15.959235 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:15.959238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:15.959239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:15.959449 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:15.959543 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:15.960264 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 07:19:15.960386 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:19:15.961884 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:15.947894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:35525 (local address 127.28.235.193:38003)
0926 07:19:15.948572 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:15.948617 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:15.949511 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:15.950095 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:15.950103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:15.950470 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:15.950965 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:15.950979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:15.951943 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:15.951952 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:15.952856 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:15.952865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:15.953483 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:15.953487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:15.953671 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:15.954330 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:15.954348 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:15.956047 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:15.957998 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:15.958007 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:15.958043 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:15.958500 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:15.958911 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:15.958915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:15.958917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:15.959124 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:15.959573 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:15.959582 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:15.959963 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 07:19:15.960212 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 07:19:15.960612 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":121,"threads_started":1}
I20260926 07:19:15.963378 29858 heartbeater.cc:347] Connected to a master server at 127.28.235.254:35525
I20260926 07:19:15.963727 29858 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:15.964349 29858 heartbeater.cc:511] Master 127.28.235.254:35525 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:19:15.966182 29667 ts_manager.cc:194] Registered new tserver with Master: cc02c3d6920542e3801d7fa404c17071 (127.28.235.193:44345)
I20260926 07:19:15.967586 29667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:38003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:19:15.980707 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35525
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:16.095769 29863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:16.096094 29863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:16.096189 29863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:16.100410 29863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:16.100519 29863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:16.100620 29863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:16.106316 29863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:16.107497 29863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:16.108500 29863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:16.118613 29871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:16.118676 29869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:16.118695 29868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:16.119127 29863 server_base.cc:1034] running on GCE node
I20260926 07:19:16.119540 29863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:16.120190 29863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:16.121353 29863 hybrid_clock.cc:648] HybridClock initialized: now 1790407156121313 us; error 48 us; skew 500 ppm
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[29622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:16.124240 29863 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:19:16.125202 29863 webserver.cc:460] Webserver started at http://127.28.235.194:36565/ using document root <none> and password file <none>
I20260926 07:19:16.125788 29863 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:16.125851 29863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:16.126014 29863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:16.128068 29863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "47debada4c34404b9c666b13071d8eae"
format_stamp: "Formatted at 2026-09-26 07:19:16 on dist-test-slave-7zlw"
server_key: "b27eb28593be68e7fef48eebc4efbf8f"
server_key_iv: "822b1a2a74071ba2fc2504c52a5adce3"
server_key_version: "encryptionkey@0"
I20260926 07:19:16.128582 29863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "47debada4c34404b9c666b13071d8eae"
format_stamp: "Formatted at 2026-09-26 07:19:16 on dist-test-slave-7zlw"
server_key: "b27eb28593be68e7fef48eebc4efbf8f"
server_key_iv: "822b1a2a74071ba2fc2504c52a5adce3"
server_key_version: "encryptionkey@0"
I20260926 07:19:16.132244 29863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 07:19:16.134889 29878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:16.136188 29863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:19:16.136361 29863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "47debada4c34404b9c666b13071d8eae"
format_stamp: "Formatted at 2026-09-26 07:19:16 on dist-test-slave-7zlw"
server_key: "b27eb28593be68e7fef48eebc4efbf8f"
server_key_iv: "822b1a2a74071ba2fc2504c52a5adce3"
server_key_version: "encryptionkey@0"
I20260926 07:19:16.136485 29863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:16.162914 29863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:16.166767 29863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:16.167006 29863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:16.167788 29863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:16.168670 29863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:16.168742 29863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:16.168809 29863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:16.168854 29863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:16.178521 29863 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:39373
I20260926 07:19:16.178543 29991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:39373 every 8 connection(s)
I20260926 07:19:16.179494 29863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:16.187891 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 29863
I20260926 07:19:16.188014 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:16.188279 29615 external_mini_cluster.cc:1428] Setting key 985498afb99442cdd4dea4c1eec595a5
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[29622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:16.194581 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:16.181543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:33801 (local address 127.28.235.254:35525)
0926 07:19:16.181727 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:16.181731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:16.182918 (+ 1187us) server_negotiation.cc:407] Connection header received
0926 07:19:16.183845 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:16.183848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:16.183897 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:16.183974 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:16.185474 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.185996 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.187230 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.187454 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.187534 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:16.190569 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:16.190576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:16.190582 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:16.190617 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:16.192203 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:16.192843 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:16.192846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:16.192847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:16.192922 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:16.193351 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:16.193353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:16.193354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:16.193536 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:16.193615 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:16.194350 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 07:19:16.194460 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 07:19:16.195914 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:16.181981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:35525 (local address 127.28.235.194:33801)
0926 07:19:16.182757 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:16.182803 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:16.183650 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:16.184206 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:16.184215 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:16.184639 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:16.185323 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:16.185342 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.186135 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.186138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:16.187121 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:16.187130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.187647 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.187651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:16.187815 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:16.188494 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:16.188512 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:16.190391 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:16.192329 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:16.192336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:16.192355 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:16.192744 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:16.193054 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:16.193060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:16.193065 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:16.193253 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:16.193646 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:16.193656 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:16.194006 (+ 350us) client_negotiation.cc:769] Sending connection context
0926 07:19:16.194277 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 07:19:16.194668 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":115,"threads_started":1}
I20260926 07:19:16.197235 29992 heartbeater.cc:347] Connected to a master server at 127.28.235.254:35525
I20260926 07:19:16.197525 29992 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:16.198043 29992 heartbeater.cc:511] Master 127.28.235.254:35525 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:19:16.199139 29667 ts_manager.cc:194] Registered new tserver with Master: 47debada4c34404b9c666b13071d8eae (127.28.235.194:39373)
I20260926 07:19:16.199671 29667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:33801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:19:16.219990 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35525
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:16.336148 29997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:16.336627 29997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:16.336720 29997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:16.345589 29997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:16.345691 29997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:16.345817 29997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:16.355834 29997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:16.357395 29997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:16.358785 29997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:19:16.371676 29997 server_base.cc:1034] running on GCE node
W20260926 07:19:16.371740 30003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:16.371713 30002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:16.371902 30005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:16.372516 29997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:16.373407 29997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:16.374609 29997 hybrid_clock.cc:648] HybridClock initialized: now 1790407156374569 us; error 62 us; skew 500 ppm
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[29622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:16.377815 29997 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:19:16.379084 29997 webserver.cc:460] Webserver started at http://127.28.235.195:41139/ using document root <none> and password file <none>
I20260926 07:19:16.379863 29997 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:16.379946 29997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:16.380165 29997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:16.382365 29997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "b4ed98a383a248e6b09cd1b9c9b5a9b5"
format_stamp: "Formatted at 2026-09-26 07:19:16 on dist-test-slave-7zlw"
server_key: "18e3988236412cb047520edb55fc1a2f"
server_key_iv: "eff355001f04c4ac09f2d730c455b4e4"
server_key_version: "encryptionkey@0"
I20260926 07:19:16.382961 29997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "b4ed98a383a248e6b09cd1b9c9b5a9b5"
format_stamp: "Formatted at 2026-09-26 07:19:16 on dist-test-slave-7zlw"
server_key: "18e3988236412cb047520edb55fc1a2f"
server_key_iv: "eff355001f04c4ac09f2d730c455b4e4"
server_key_version: "encryptionkey@0"
I20260926 07:19:16.386786 29997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:19:16.389292 30012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:16.390591 29997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:19:16.390744 29997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "b4ed98a383a248e6b09cd1b9c9b5a9b5"
format_stamp: "Formatted at 2026-09-26 07:19:16 on dist-test-slave-7zlw"
server_key: "18e3988236412cb047520edb55fc1a2f"
server_key_iv: "eff355001f04c4ac09f2d730c455b4e4"
server_key_version: "encryptionkey@0"
I20260926 07:19:16.390885 29997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:16.419893 29997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:16.423609 29997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:16.423848 29997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:16.424511 29997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:16.425449 29997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:16.425524 29997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:16.425598 29997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:16.425657 29997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:16.435123 29997 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:36031
I20260926 07:19:16.435177 30125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:36031 every 8 connection(s)
I20260926 07:19:16.436107 29997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:16.437146 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 29997
I20260926 07:19:16.437278 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:16.437537 29615 external_mini_cluster.cc:1428] Setting key 32c9b2a81c6b069a6d7824f17fd63005
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[29622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407156, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:16.451318 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:16.438218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:55609 (local address 127.28.235.254:35525)
0926 07:19:16.438349 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:16.438353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:16.439660 (+ 1307us) server_negotiation.cc:407] Connection header received
0926 07:19:16.440961 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:16.440964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:16.441036 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:16.441103 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:16.442394 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.442914 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.444129 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.444360 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.444447 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:16.447280 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:16.447283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:16.447285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:16.447314 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:16.448847 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:16.449480 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:16.449484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:16.449485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:16.449577 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:16.449950 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:16.449954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:16.449955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:16.450254 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:16.450366 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:16.450930 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 07:19:16.451094 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:19:16.452350 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:16.438668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:35525 (local address 127.28.235.195:55609)
0926 07:19:16.439511 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:16.439559 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:16.440718 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:16.441272 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:16.441279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:16.441657 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:16.442262 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:16.442276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.443025 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.443028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:16.444006 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:16.444016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.444529 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.444532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:16.444687 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:16.445427 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:16.445444 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:16.447062 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:16.448972 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:16.448977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:16.448992 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:16.449359 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:16.449669 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:16.449672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:16.449673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:16.449833 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:16.450371 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:16.450377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:16.450636 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 07:19:16.450862 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 07:19:16.451211 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":123,"threads_started":1}
I20260926 07:19:16.453485 30126 heartbeater.cc:347] Connected to a master server at 127.28.235.254:35525
I20260926 07:19:16.453754 30126 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:16.454273 30126 heartbeater.cc:511] Master 127.28.235.254:35525 requested a full tablet report, sending...
I20260926 07:19:16.455431 29667 ts_manager.cc:194] Registered new tserver with Master: b4ed98a383a248e6b09cd1b9c9b5a9b5 (127.28.235.195:36031)
I20260926 07:19:16.456084 29667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:55609
I20260926 07:19:16.465310 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[29622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407156, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[29622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407156, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:19:16.493002 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:16.483005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.28.235.193:44345)
0926 07:19:16.483338 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:16.483344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:16.483369 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:19:16.483491 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:16.483497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:16.483698 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:16.483833 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:16.484515 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.485343 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.486169 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.486583 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.486668 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:16.489320 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:16.489330 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:16.489345 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:16.489370 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:16.491138 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:16.491654 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:16.491661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:16.491663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:16.491759 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:16.492027 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:16.492030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:16.492031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:16.492291 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:16.492450 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:16.492613 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:19:16.492793 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":88,"threads_started":1}
W20260926 07:19:16.494124 29772 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44772
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[29622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[29622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:19:16.522150 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:16.512201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.28.235.193:44345)
0926 07:19:16.512407 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:16.512412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:16.512591 (+ 179us) server_negotiation.cc:407] Connection header received
0926 07:19:16.512780 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:16.512789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:16.512837 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:16.512936 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:16.513810 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.514374 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.515356 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:16.515613 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:16.515731 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:16.518734 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:16.518739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:16.518741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:16.518772 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:16.520229 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:16.520750 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:16.520755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:16.520757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:16.520851 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:16.521239 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:16.521247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:16.521249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:16.521435 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:16.521583 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:16.521842 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 07:19:16.521969 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 07:19:16.523370 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 29729
I20260926 07:19:16.529969 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 29863
I20260926 07:19:16.538616 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 29997
I20260926 07:19:16.544512 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 29635
2026-09-26T07:19:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2352 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[30140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[30140](info): set up 2 sockets
Sep 26 07:19:16 dist-test-slave-7zlw krb5kdc[30140](info): commencing operation
krb5kdc: starting...
W20260926 07:19:17.592623 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:17 dist-test-slave-7zlw krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:17.718772 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34937
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:34937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:17.830564 30153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:17.830849 30153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:17.830892 30153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:17.835027 30153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:17.835088 30153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:17.835107 30153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:17.835124 30153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:17.835194 30153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:17.840638 30153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:34937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:17.841796 30153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:17.842927 30153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:17.851634 30158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:17.851650 30159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:17.851650 30161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:17.851876 30153 server_base.cc:1034] running on GCE node
I20260926 07:19:17.852602 30153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:17.853751 30153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:17.854939 30153 hybrid_clock.cc:648] HybridClock initialized: now 1790407157854909 us; error 45 us; skew 500 ppm
Sep 26 07:19:17 dist-test-slave-7zlw krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407157, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:17.857930 30153 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:17.858948 30153 webserver.cc:460] Webserver started at http://127.28.235.254:43259/ using document root <none> and password file <none>
I20260926 07:19:17.859534 30153 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:17.859601 30153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:17.859815 30153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:17.861997 30153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "864a1850fd4f44279202d712fb332d29"
format_stamp: "Formatted at 2026-09-26 07:19:17 on dist-test-slave-7zlw"
server_key: "a1bb3ec15b4f6a94c9fe6bb8951ff98c"
server_key_iv: "6fd459a654b72e1e1a1445e228c081de"
server_key_version: "encryptionkey@0"
I20260926 07:19:17.862599 30153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "864a1850fd4f44279202d712fb332d29"
format_stamp: "Formatted at 2026-09-26 07:19:17 on dist-test-slave-7zlw"
server_key: "a1bb3ec15b4f6a94c9fe6bb8951ff98c"
server_key_iv: "6fd459a654b72e1e1a1445e228c081de"
server_key_version: "encryptionkey@0"
I20260926 07:19:17.866508 30153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:17.869222 30168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:17.870530 30153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:17.870664 30153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "864a1850fd4f44279202d712fb332d29"
format_stamp: "Formatted at 2026-09-26 07:19:17 on dist-test-slave-7zlw"
server_key: "a1bb3ec15b4f6a94c9fe6bb8951ff98c"
server_key_iv: "6fd459a654b72e1e1a1445e228c081de"
server_key_version: "encryptionkey@0"
I20260926 07:19:17.870781 30153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:17.886493 30153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:17.889705 30153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:17.889928 30153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:17.897759 30153 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:34937
I20260926 07:19:17.897789 30220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:34937 every 8 connection(s)
I20260926 07:19:17.898836 30153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:17.901947 30221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:17.905514 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 30153
I20260926 07:19:17.905644 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:17.905921 29615 external_mini_cluster.cc:1428] Setting key 8b9114eb716540bee3d44192bf35d3a6
I20260926 07:19:17.908502 30221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29: Bootstrap starting.
I20260926 07:19:17.911319 30221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:17.912317 30221 log.cc:824] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:17.914971 30221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29: No bootstrap required, opened a new log
Sep 26 07:19:17 dist-test-slave-7zlw krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:17.918329 30221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a1850fd4f44279202d712fb332d29" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34937 } }
I20260926 07:19:17.918615 30221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:17.918699 30221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864a1850fd4f44279202d712fb332d29, State: Initialized, Role: FOLLOWER
I20260926 07:19:17.919168 30221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a1850fd4f44279202d712fb332d29" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34937 } }
I20260926 07:19:17.919411 30221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:17.919518 30221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:17.919637 30221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:17.920996 30221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a1850fd4f44279202d712fb332d29" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34937 } }
I20260926 07:19:17.921423 30221 leader_election.cc:302] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 864a1850fd4f44279202d712fb332d29; no voters:
I20260926 07:19:17.921846 30221 leader_election.cc:288] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:17.922315 30227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:17.922632 30227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [term 1 LEADER]: Becoming Leader. State: Replica: 864a1850fd4f44279202d712fb332d29, State: Running, Role: LEADER
I20260926 07:19:17.923051 30227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a1850fd4f44279202d712fb332d29" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34937 } }
I20260926 07:19:17.923308 30221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:17.924315 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:17.907480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.28.235.254:34937)
0926 07:19:17.907944 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:17.907960 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:17.907998 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:19:17.908716 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:17.908735 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:17.909174 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:17.909481 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:17.910444 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:17.911562 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:17.912726 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:17.913480 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:17.913609 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:17.916493 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:17.916498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:17.916514 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:17.916544 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:17.919217 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:17.920078 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:17.920091 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:17.920101 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:17.920300 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:17.920664 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:17.920668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:17.920670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:17.921366 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:17.921590 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:17.922644 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0926 07:19:17.923075 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":154,"threads_started":1}
I20260926 07:19:17.924367 30227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 864a1850fd4f44279202d712fb332d29. Latest consensus state: current_term: 1 leader_uuid: "864a1850fd4f44279202d712fb332d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a1850fd4f44279202d712fb332d29" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34937 } } }
I20260926 07:19:17.924826 30226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "864a1850fd4f44279202d712fb332d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864a1850fd4f44279202d712fb332d29" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34937 } } }
I20260926 07:19:17.924926 30226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:17.924460 30227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29 [sys.catalog]: This master's current role is: LEADER
W20260926 07:19:17.928414 30241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:19:17.928514 30241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:19:17.928584 30239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:17.929450 30239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:17.935426 30239 catalog_manager.cc:1347] Generated new cluster ID: 13ef6fb900e94538a858032484a1afd3
I20260926 07:19:17.935508 30239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:17.943683 30239 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:17.944978 30239 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:17.950277 30239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 864a1850fd4f44279202d712fb332d29: Generated new TSK 0
I20260926 07:19:17.951151 30239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:19:17.988313 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34937
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:18.125084 30247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:18.125367 30247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:18.125456 30247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:18.129683 30247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:18.129772 30247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:18.129873 30247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:18.135396 30247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34937
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:18.136500 30247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:18.137620 30247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:18.147552 30253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:18.147540 30252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:18.147540 30255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:18.147902 30247 server_base.cc:1034] running on GCE node
I20260926 07:19:18.148404 30247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:18.149056 30247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:18.150290 30247 hybrid_clock.cc:648] HybridClock initialized: now 1790407158150270 us; error 31 us; skew 500 ppm
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:18.153407 30247 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:19:18.154387 30247 webserver.cc:460] Webserver started at http://127.28.235.193:44671/ using document root <none> and password file <none>
I20260926 07:19:18.155000 30247 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:18.155095 30247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:18.155385 30247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:18.157651 30247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "ed642bc008f842948f7603aa64c1e7e6"
format_stamp: "Formatted at 2026-09-26 07:19:18 on dist-test-slave-7zlw"
server_key: "312fdd830dee8b13a7fa84b57705ab2a"
server_key_iv: "f3bf852187afbbd824e45ebcbbc6e32a"
server_key_version: "encryptionkey@0"
I20260926 07:19:18.158277 30247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "ed642bc008f842948f7603aa64c1e7e6"
format_stamp: "Formatted at 2026-09-26 07:19:18 on dist-test-slave-7zlw"
server_key: "312fdd830dee8b13a7fa84b57705ab2a"
server_key_iv: "f3bf852187afbbd824e45ebcbbc6e32a"
server_key_version: "encryptionkey@0"
I20260926 07:19:18.162190 30247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 07:19:18.164860 30262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:18.166600 30247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:19:18.166775 30247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "ed642bc008f842948f7603aa64c1e7e6"
format_stamp: "Formatted at 2026-09-26 07:19:18 on dist-test-slave-7zlw"
server_key: "312fdd830dee8b13a7fa84b57705ab2a"
server_key_iv: "f3bf852187afbbd824e45ebcbbc6e32a"
server_key_version: "encryptionkey@0"
I20260926 07:19:18.166909 30247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:18.185384 30247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:18.188942 30247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:18.189203 30247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:18.189913 30247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:18.190834 30247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:18.190913 30247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:18.190985 30247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:18.191041 30247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:18.200783 30247 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:46139
I20260926 07:19:18.200803 30375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:46139 every 8 connection(s)
I20260926 07:19:18.201803 30247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:18.206338 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 30247
I20260926 07:19:18.206489 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:18.206822 29615 external_mini_cluster.cc:1428] Setting key 1b05f7a927c4a1398dd0ae9f5d2f8100
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:19:18.218983 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.203972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:43681 (local address 127.28.235.254:34937)
0926 07:19:18.204148 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.204154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.205165 (+ 1011us) server_negotiation.cc:407] Connection header received
0926 07:19:18.206668 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.206673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.206736 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.206834 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:18.208309 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.209198 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.211037 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.211320 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.211415 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.214740 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:18.214745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:18.214748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:18.214777 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:18.216477 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.217148 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.217153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.217155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.217278 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.217714 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.217717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.217719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.217930 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:18.218051 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.218636 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.218796 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:19:18.219918 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.204443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34937 (local address 127.28.235.193:43681)
0926 07:19:18.204921 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:18.204970 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:18.206393 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:18.207049 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:18.207056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:18.207556 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:18.208126 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.208149 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.209385 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.209388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.210890 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.210905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.211497 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.211504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.211828 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.212586 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:18.212615 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:18.214530 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:18.216619 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:18.216626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:18.216645 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:18.216978 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:18.217412 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:18.217416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:18.217418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:18.217593 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:18.218052 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:18.218060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:18.218362 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 07:19:18.218577 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 07:19:18.218987 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:19:18.221967 30376 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34937
I20260926 07:19:18.222296 30376 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:18.222868 30376 heartbeater.cc:511] Master 127.28.235.254:34937 requested a full tablet report, sending...
I20260926 07:19:18.224757 30185 ts_manager.cc:194] Registered new tserver with Master: ed642bc008f842948f7603aa64c1e7e6 (127.28.235.193:46139)
I20260926 07:19:18.226163 30185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:43681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:19:18.240715 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34937
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:18.364928 30381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:18.365356 30381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:18.365438 30381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:18.369764 30381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:18.369823 30381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:18.369911 30381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:18.375556 30381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34937
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:18.376677 30381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:18.377816 30381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:18.387952 30386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:18.388000 30381 server_base.cc:1034] running on GCE node
W20260926 07:19:18.388012 30389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:18.387940 30387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:18.388669 30381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:18.389422 30381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:18.390645 30381 hybrid_clock.cc:648] HybridClock initialized: now 1790407158390592 us; error 62 us; skew 500 ppm
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:18.393445 30381 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:19:18.394307 30381 webserver.cc:460] Webserver started at http://127.28.235.194:37051/ using document root <none> and password file <none>
I20260926 07:19:18.394871 30381 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:18.394958 30381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:18.395196 30381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:18.397353 30381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "4b3c874771ea463e966ebf2947de1b96"
format_stamp: "Formatted at 2026-09-26 07:19:18 on dist-test-slave-7zlw"
server_key: "2224294724c40fd3636e472ca0d2a77c"
server_key_iv: "f31886bef2df23faa2f53ec8833c2c55"
server_key_version: "encryptionkey@0"
I20260926 07:19:18.397936 30381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "4b3c874771ea463e966ebf2947de1b96"
format_stamp: "Formatted at 2026-09-26 07:19:18 on dist-test-slave-7zlw"
server_key: "2224294724c40fd3636e472ca0d2a77c"
server_key_iv: "f31886bef2df23faa2f53ec8833c2c55"
server_key_version: "encryptionkey@0"
I20260926 07:19:18.401789 30381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:19:18.404281 30396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:18.405640 30381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:18.405793 30381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "4b3c874771ea463e966ebf2947de1b96"
format_stamp: "Formatted at 2026-09-26 07:19:18 on dist-test-slave-7zlw"
server_key: "2224294724c40fd3636e472ca0d2a77c"
server_key_iv: "f31886bef2df23faa2f53ec8833c2c55"
server_key_version: "encryptionkey@0"
I20260926 07:19:18.405925 30381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:18.427187 30381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:18.430547 30381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:18.430765 30381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:18.431380 30381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:18.432237 30381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:18.432305 30381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:18.432400 30381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:18.432443 30381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:18.441609 30381 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:33803
I20260926 07:19:18.441628 30509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:33803 every 8 connection(s)
I20260926 07:19:18.442560 30381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:18.448030 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 30381
I20260926 07:19:18.448123 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:18.448518 29615 external_mini_cluster.cc:1428] Setting key 080e036d0eee25f949446d068af88d56
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:18.457226 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.444482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:45081 (local address 127.28.235.254:34937)
0926 07:19:18.444625 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.444629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.445543 (+ 914us) server_negotiation.cc:407] Connection header received
0926 07:19:18.446891 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.446895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.446955 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.447082 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:18.448368 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.449085 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.450639 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.450880 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.450993 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.453556 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:18.453559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:18.453562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:18.453595 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:18.455077 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.455625 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.455629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.455631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.455712 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.456068 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.456073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.456075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.456270 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:18.456377 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.456883 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.457057 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:19:18.458091 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.444882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34937 (local address 127.28.235.194:45081)
0926 07:19:18.445392 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:18.445437 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:18.446660 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:18.447259 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:18.447266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:18.447636 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:18.448208 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.448229 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.449205 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.449210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.450516 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.450526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.451037 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.451041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.451220 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.451864 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:18.451883 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:18.453381 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:18.455217 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:18.455223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:18.455247 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:18.455514 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:18.455813 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:18.455817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:18.455820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:18.455971 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:18.456376 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:18.456383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:18.456644 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 07:19:18.456843 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 07:19:18.457220 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":163,"threads_started":1}
I20260926 07:19:18.459575 30510 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34937
I20260926 07:19:18.459851 30510 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
I20260926 07:19:18.460462 30510 heartbeater.cc:511] Master 127.28.235.254:34937 requested a full tablet report, sending...
I20260926 07:19:18.461757 30185 ts_manager.cc:194] Registered new tserver with Master: 4b3c874771ea463e966ebf2947de1b96 (127.28.235.194:33803)
I20260926 07:19:18.462358 30185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:45081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:19:18.477324 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34937
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:18.593833 30515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:18.594123 30515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:18.594177 30515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:18.598232 30515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:18.598299 30515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:18.598362 30515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:18.603777 30515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34937
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:18.604889 30515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:18.606009 30515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:19:18.616108 30515 server_base.cc:1034] running on GCE node
W20260926 07:19:18.616055 30520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:18.616070 30521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:18.616055 30523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:18.616834 30515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:18.617558 30515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:18.618721 30515 hybrid_clock.cc:648] HybridClock initialized: now 1790407158618686 us; error 49 us; skew 500 ppm
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:18.621759 30515 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:19:18.622643 30515 webserver.cc:460] Webserver started at http://127.28.235.195:34135/ using document root <none> and password file <none>
I20260926 07:19:18.623268 30515 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:18.623333 30515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:18.623505 30515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:18.625932 30515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "bdddc249a341433aafec0983ebef1580"
format_stamp: "Formatted at 2026-09-26 07:19:18 on dist-test-slave-7zlw"
server_key: "d1ed58ce5bf4f48e456623c73e604f5d"
server_key_iv: "393fbb7143a4a723ebe885401f291445"
server_key_version: "encryptionkey@0"
I20260926 07:19:18.626487 30515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "bdddc249a341433aafec0983ebef1580"
format_stamp: "Formatted at 2026-09-26 07:19:18 on dist-test-slave-7zlw"
server_key: "d1ed58ce5bf4f48e456623c73e604f5d"
server_key_iv: "393fbb7143a4a723ebe885401f291445"
server_key_version: "encryptionkey@0"
I20260926 07:19:18.630546 30515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:18.633131 30530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:18.634475 30515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:18.634624 30515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "bdddc249a341433aafec0983ebef1580"
format_stamp: "Formatted at 2026-09-26 07:19:18 on dist-test-slave-7zlw"
server_key: "d1ed58ce5bf4f48e456623c73e604f5d"
server_key_iv: "393fbb7143a4a723ebe885401f291445"
server_key_version: "encryptionkey@0"
I20260926 07:19:18.634753 30515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:18.653234 30515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:18.656759 30515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:18.657003 30515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:18.657681 30515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:18.658638 30515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:18.658716 30515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:18.658804 30515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:18.658850 30515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:18.668900 30515 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:35611
I20260926 07:19:18.668919 30643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:35611 every 8 connection(s)
I20260926 07:19:18.669934 30515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:18.674304 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 30515
I20260926 07:19:18.674438 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:18.674762 29615 external_mini_cluster.cc:1428] Setting key fbc772e471dedea46f4c09ed144a6577
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407158, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:18.684831 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.671999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:42427 (local address 127.28.235.254:34937)
0926 07:19:18.672147 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.672151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.673120 (+ 969us) server_negotiation.cc:407] Connection header received
0926 07:19:18.674712 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.674716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.674788 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.674891 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:18.676347 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.677067 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.678336 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.678568 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.678658 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.681396 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:18.681400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:18.681402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:18.681428 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:18.682962 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.683463 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.683465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.683466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.683534 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.683823 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.683825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.683826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.683994 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:18.684067 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.684586 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.684708 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:19:18.685714 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.672478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34937 (local address 127.28.235.195:42427)
0926 07:19:18.672935 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:18.672980 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:18.674416 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:18.675083 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:18.675091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:18.675614 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:18.676192 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.676213 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.677196 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.677201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.678195 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.678205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.678707 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.678709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.678856 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.679592 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:18.679609 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:18.681206 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:18.683063 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:18.683068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:18.683081 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:18.683374 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:18.683611 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:18.683613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:18.683614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:18.683746 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:18.684079 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:18.684084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:18.684361 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 07:19:18.684527 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 07:19:18.684798 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":103,"threads_started":1}
I20260926 07:19:18.686905 30644 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34937
I20260926 07:19:18.687196 30644 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:18.687834 30644 heartbeater.cc:511] Master 127.28.235.254:34937 requested a full tablet report, sending...
I20260926 07:19:18.689237 30185 ts_manager.cc:194] Registered new tserver with Master: bdddc249a341433aafec0983ebef1580 (127.28.235.195:35611)
I20260926 07:19:18.689850 30185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:42427
I20260926 07:19:18.701253 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:19:18.713753 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.705559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.28.235.254:34937)
0926 07:19:18.705704 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.705709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.705916 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:19:18.706104 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.706107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.706156 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.706247 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:18.706983 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.707558 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.708504 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.708742 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.708837 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.710491 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:18.710494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:18.710496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:18.710522 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:18.712156 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.712596 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.712599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.712600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.712680 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.712923 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.712925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.712926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.713210 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:18.713311 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.713533 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.713628 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:19:18.719079 30185 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:19:18.722309 30185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:18.738013 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.731835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.28.235.193:46139)
0926 07:19:18.732896 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.732904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.732927 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:19:18.733005 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.733011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.733292 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.733452 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:18.733773 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.734721 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.736325 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.737229 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.737384 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.737531 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.737733 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.737848 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":99,"threads_started":1}
I20260926 07:19:18.738082 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.732666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.28.235.194:33803)
0926 07:19:18.733003 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.733035 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.733793 (+ 758us) server_negotiation.cc:407] Connection header received
0926 07:19:18.733884 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.733889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.734066 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.734228 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:18.734462 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.735435 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.736819 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.737565 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.737708 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.737828 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.737873 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.737948 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":133,"threads_started":1}
I20260926 07:19:18.738189 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.732473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:33803 (local address 127.0.0.1:41342)
0926 07:19:18.733675 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:18.733690 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:18.733787 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:18.734230 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:18.734233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:18.734251 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:18.734347 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.734353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.735565 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.735571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.736696 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.736706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.737681 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.737684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.737764 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.737793 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:19:18.737848 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 07:19:18.738034 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"mutex_wait_us":129,"thread_start_us":87,"threads_started":1}
I20260926 07:19:18.738319 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.732092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.28.235.195:35611)
0926 07:19:18.732929 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.732942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.732967 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:19:18.733071 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.733079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.733297 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.733453 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:18.733918 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.735104 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.736506 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.737299 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.737404 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.737531 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.738065 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.738168 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":95,"threads_started":1}
I20260926 07:19:18.738909 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.731709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:46139 (local address 127.0.0.1:54096)
0926 07:19:18.732346 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:18.732385 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:18.732669 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:18.733440 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:18.733446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:18.733468 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:18.733618 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.733627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.734835 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.734838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.736182 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.736192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.737356 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.737360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.737477 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.737522 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 07:19:18.738728 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0926 07:19:18.738811 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":98,"threads_started":1}
I20260926 07:19:18.739431 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.731748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:35611 (local address 127.0.0.1:57896)
0926 07:19:18.732463 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:18.732476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:18.732604 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:18.733643 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:18.733648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:18.733667 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:18.733784 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.733791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.735240 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.735244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.736381 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.736390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.737601 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.737604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.737724 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.737753 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:19:18.737827 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:19:18.739311 (+ 1484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":1284,"thread_start_us":99,"threads_started":1}
I20260926 07:19:18.741694 30310 tablet_service.cc:1467] Processing CreateTablet for tablet f6435a38eeb34b608d4eb4f7ec84cb88 (DEFAULT_TABLE table=test-table [id=4eb6aa67977c498895b4a59cf644e907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:18.741698 30444 tablet_service.cc:1467] Processing CreateTablet for tablet f6435a38eeb34b608d4eb4f7ec84cb88 (DEFAULT_TABLE table=test-table [id=4eb6aa67977c498895b4a59cf644e907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:18.742027 30578 tablet_service.cc:1467] Processing CreateTablet for tablet f6435a38eeb34b608d4eb4f7ec84cb88 (DEFAULT_TABLE table=test-table [id=4eb6aa67977c498895b4a59cf644e907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:18.742962 30578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6435a38eeb34b608d4eb4f7ec84cb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:18.743000 30444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6435a38eeb34b608d4eb4f7ec84cb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:18.743000 30310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6435a38eeb34b608d4eb4f7ec84cb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:18.750005 30661 tablet_bootstrap.cc:492] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580: Bootstrap starting.
I20260926 07:19:18.750985 30663 tablet_bootstrap.cc:492] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96: Bootstrap starting.
I20260926 07:19:18.752266 30661 tablet_bootstrap.cc:654] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:18.753165 30663 tablet_bootstrap.cc:654] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:18.753439 30662 tablet_bootstrap.cc:492] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6: Bootstrap starting.
I20260926 07:19:18.753621 30661 log.cc:824] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:18.754036 30663 log.cc:824] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:18.755792 30662 tablet_bootstrap.cc:654] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:18.756067 30661 tablet_bootstrap.cc:492] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580: No bootstrap required, opened a new log
I20260926 07:19:18.756270 30661 ts_tablet_manager.cc:1397] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:19:18.756636 30662 log.cc:824] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:18.756794 30663 tablet_bootstrap.cc:492] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96: No bootstrap required, opened a new log
I20260926 07:19:18.757047 30663 ts_tablet_manager.cc:1397] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260926 07:19:18.758620 30662 tablet_bootstrap.cc:492] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6: No bootstrap required, opened a new log
I20260926 07:19:18.758807 30662 ts_tablet_manager.cc:1397] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:19:18.760120 30663 raft_consensus.cc:348] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed642bc008f842948f7603aa64c1e7e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46139 } } peers { permanent_uuid: "bdddc249a341433aafec0983ebef1580" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35611 } } peers { permanent_uuid: "4b3c874771ea463e966ebf2947de1b96" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33803 } }
I20260926 07:19:18.760397 30663 raft_consensus.cc:374] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:18.760457 30663 raft_consensus.cc:729] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3c874771ea463e966ebf2947de1b96, State: Initialized, Role: FOLLOWER
I20260926 07:19:18.760273 30661 raft_consensus.cc:348] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed642bc008f842948f7603aa64c1e7e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46139 } } peers { permanent_uuid: "bdddc249a341433aafec0983ebef1580" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35611 } } peers { permanent_uuid: "4b3c874771ea463e966ebf2947de1b96" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33803 } }
I20260926 07:19:18.760602 30661 raft_consensus.cc:374] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:18.760671 30661 raft_consensus.cc:729] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdddc249a341433aafec0983ebef1580, State: Initialized, Role: FOLLOWER
I20260926 07:19:18.760950 30663 consensus_queue.cc:260] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed642bc008f842948f7603aa64c1e7e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46139 } } peers { permanent_uuid: "bdddc249a341433aafec0983ebef1580" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35611 } } peers { permanent_uuid: "4b3c874771ea463e966ebf2947de1b96" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33803 } }
I20260926 07:19:18.761145 30661 consensus_queue.cc:260] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed642bc008f842948f7603aa64c1e7e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46139 } } peers { permanent_uuid: "bdddc249a341433aafec0983ebef1580" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35611 } } peers { permanent_uuid: "4b3c874771ea463e966ebf2947de1b96" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33803 } }
I20260926 07:19:18.761693 30510 heartbeater.cc:503] Master 127.28.235.254:34937 was elected leader, sending a full tablet report...
I20260926 07:19:18.762070 30663 ts_tablet_manager.cc:1428] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260926 07:19:18.762106 30661 ts_tablet_manager.cc:1428] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:19:18.762431 30644 heartbeater.cc:503] Master 127.28.235.254:34937 was elected leader, sending a full tablet report...
I20260926 07:19:18.763290 30662 raft_consensus.cc:348] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed642bc008f842948f7603aa64c1e7e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46139 } } peers { permanent_uuid: "bdddc249a341433aafec0983ebef1580" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35611 } } peers { permanent_uuid: "4b3c874771ea463e966ebf2947de1b96" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33803 } }
I20260926 07:19:18.763557 30662 raft_consensus.cc:374] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:18.763615 30662 raft_consensus.cc:729] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed642bc008f842948f7603aa64c1e7e6, State: Initialized, Role: FOLLOWER
I20260926 07:19:18.763998 30662 consensus_queue.cc:260] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed642bc008f842948f7603aa64c1e7e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46139 } } peers { permanent_uuid: "bdddc249a341433aafec0983ebef1580" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35611 } } peers { permanent_uuid: "4b3c874771ea463e966ebf2947de1b96" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33803 } }
I20260926 07:19:18.764756 30376 heartbeater.cc:503] Master 127.28.235.254:34937 was elected leader, sending a full tablet report...
I20260926 07:19:18.764889 30662 ts_tablet_manager.cc:1428] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260926 07:19:18.784471 30668 raft_consensus.cc:482] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:18.784721 30668 raft_consensus.cc:504] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed642bc008f842948f7603aa64c1e7e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46139 } } peers { permanent_uuid: "bdddc249a341433aafec0983ebef1580" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35611 } } peers { permanent_uuid: "4b3c874771ea463e966ebf2947de1b96" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33803 } }
I20260926 07:19:18.786048 30668 leader_election.cc:288] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed642bc008f842948f7603aa64c1e7e6 (127.28.235.193:46139), 4b3c874771ea463e966ebf2947de1b96 (127.28.235.194:33803)
I20260926 07:19:18.789624 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.786437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:46695 (local address 127.28.235.193:46139)
0926 07:19:18.786583 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.786587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.786602 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:19:18.786659 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.786665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.786722 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.786820 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:18.787110 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.787619 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.788637 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.789075 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.789211 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.789270 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.789430 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.789495 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:19:18.789624 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.786228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:46139 (local address 127.28.235.195:46695)
0926 07:19:18.786356 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:18.786375 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:18.786469 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:18.786838 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:18.786841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:18.786883 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:18.786993 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.786998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.787753 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.787758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.788515 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.788523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.789196 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.789200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.789312 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.789329 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 07:19:18.789416 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:19:18.789489 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"spinlock_wait_cycles":28800}
I20260926 07:19:18.790002 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.786519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:38817 (local address 127.28.235.194:33803)
0926 07:19:18.786643 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.786648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.786833 (+ 185us) server_negotiation.cc:407] Connection header received
0926 07:19:18.787003 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.787006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.787057 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.787144 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:18.787403 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.787888 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.788897 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.789384 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.789512 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.789560 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.789809 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.789892 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:19:18.790359 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.786404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:33803 (local address 127.28.235.195:38817)
0926 07:19:18.786743 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:18.786753 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:18.786898 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:18.787152 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:18.787156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:18.787171 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:18.787300 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.787305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.788002 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.788006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.788774 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:18.788782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.789505 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.789509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:18.789608 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.789630 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:19:18.790149 (+ 519us) client_negotiation.cc:240] Negotiation successful
0926 07:19:18.790224 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"spinlock_wait_cycles":2432,"thread_start_us":123,"threads_started":1}
I20260926 07:19:18.790395 30330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6435a38eeb34b608d4eb4f7ec84cb88" candidate_uuid: "bdddc249a341433aafec0983ebef1580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed642bc008f842948f7603aa64c1e7e6" is_pre_election: true
I20260926 07:19:18.790740 30330 raft_consensus.cc:2454] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdddc249a341433aafec0983ebef1580 in term 0.
I20260926 07:19:18.791013 30464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6435a38eeb34b608d4eb4f7ec84cb88" candidate_uuid: "bdddc249a341433aafec0983ebef1580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3c874771ea463e966ebf2947de1b96" is_pre_election: true
I20260926 07:19:18.791301 30531 leader_election.cc:302] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdddc249a341433aafec0983ebef1580, ed642bc008f842948f7603aa64c1e7e6; no voters:
I20260926 07:19:18.791399 30464 raft_consensus.cc:2454] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdddc249a341433aafec0983ebef1580 in term 0.
I20260926 07:19:18.791591 30668 raft_consensus.cc:2781] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:18.791677 30668 raft_consensus.cc:482] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:18.791710 30668 raft_consensus.cc:3037] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:18.792795 30668 raft_consensus.cc:504] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed642bc008f842948f7603aa64c1e7e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46139 } } peers { permanent_uuid: "bdddc249a341433aafec0983ebef1580" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35611 } } peers { permanent_uuid: "4b3c874771ea463e966ebf2947de1b96" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33803 } }
I20260926 07:19:18.793219 30668 leader_election.cc:288] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [CANDIDATE]: Term 1 election: Requested vote from peers ed642bc008f842948f7603aa64c1e7e6 (127.28.235.193:46139), 4b3c874771ea463e966ebf2947de1b96 (127.28.235.194:33803)
I20260926 07:19:18.793547 30330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6435a38eeb34b608d4eb4f7ec84cb88" candidate_uuid: "bdddc249a341433aafec0983ebef1580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed642bc008f842948f7603aa64c1e7e6"
I20260926 07:19:18.793650 30464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6435a38eeb34b608d4eb4f7ec84cb88" candidate_uuid: "bdddc249a341433aafec0983ebef1580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3c874771ea463e966ebf2947de1b96"
I20260926 07:19:18.793715 30330 raft_consensus.cc:3037] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:18.793816 30464 raft_consensus.cc:3037] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:18.795142 30330 raft_consensus.cc:2454] T f6435a38eeb34b608d4eb4f7ec84cb88 P ed642bc008f842948f7603aa64c1e7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdddc249a341433aafec0983ebef1580 in term 1.
I20260926 07:19:18.795140 30464 raft_consensus.cc:2454] T f6435a38eeb34b608d4eb4f7ec84cb88 P 4b3c874771ea463e966ebf2947de1b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdddc249a341433aafec0983ebef1580 in term 1.
I20260926 07:19:18.795451 30533 leader_election.cc:302] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3c874771ea463e966ebf2947de1b96, bdddc249a341433aafec0983ebef1580; no voters:
I20260926 07:19:18.795697 30668 raft_consensus.cc:2781] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:18.795948 30668 raft_consensus.cc:686] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [term 1 LEADER]: Becoming Leader. State: Replica: bdddc249a341433aafec0983ebef1580, State: Running, Role: LEADER
I20260926 07:19:18.796350 30668 consensus_queue.cc:237] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed642bc008f842948f7603aa64c1e7e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46139 } } peers { permanent_uuid: "bdddc249a341433aafec0983ebef1580" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35611 } } peers { permanent_uuid: "4b3c874771ea463e966ebf2947de1b96" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33803 } }
I20260926 07:19:18.799605 30184 catalog_manager.cc:5495] T f6435a38eeb34b608d4eb4f7ec84cb88 P bdddc249a341433aafec0983ebef1580 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdddc249a341433aafec0983ebef1580 (127.28.235.195). New cstate: current_term: 1 leader_uuid: "bdddc249a341433aafec0983ebef1580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed642bc008f842948f7603aa64c1e7e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdddc249a341433aafec0983ebef1580" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b3c874771ea463e966ebf2947de1b96" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33803 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:19:18.821882 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.812432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.28.235.193:46139)
0926 07:19:18.812610 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.812615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.812777 (+ 162us) server_negotiation.cc:407] Connection header received
0926 07:19:18.812942 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.812945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.812999 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.813145 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:18.813833 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.814360 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.815252 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.815480 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.815594 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.818300 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:18.818315 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:18.818330 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:18.818360 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:18.819935 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.820428 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.820435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.820436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.820523 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.820833 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.820837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.820839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.821187 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:18.821422 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.821580 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.821713 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:19:18.851464 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.841503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.28.235.193:46139)
0926 07:19:18.841678 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.841683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.841831 (+ 148us) server_negotiation.cc:407] Connection header received
0926 07:19:18.842021 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.842024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.842068 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.842157 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:18.842852 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.843413 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.844275 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.844507 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.844621 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.847782 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:18.847786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:18.847789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:18.847820 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:18.849545 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.849992 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.849995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.849996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.850135 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.850415 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.850418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.850419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.850624 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:18.850731 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.851080 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.851286 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:19:18.881179 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:18.871642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.28.235.193:46139)
0926 07:19:18.871911 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:18.871915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:18.872048 (+ 133us) server_negotiation.cc:407] Connection header received
0926 07:19:18.872140 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:18.872142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:18.872192 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:18.872275 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:18.873163 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.873776 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.874809 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:18.875065 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:18.875148 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:18.878014 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:18.878017 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:18.878019 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:18.878047 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:18.879470 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.879910 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.879913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.879914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.879993 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:18.880268 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:18.880271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:18.880272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:18.880445 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:18.880549 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:18.880830 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 07:19:18.880966 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
W20260926 07:19:18.882826 30290 tablet_service.cc:2908] Rejecting scan request for tablet f6435a38eeb34b608d4eb4f7ec84cb88: Uninitialized: clean time has not yet been initialized
I20260926 07:19:18.885435 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 30247
I20260926 07:19:18.891747 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 30381
I20260926 07:19:18.898375 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 30515
I20260926 07:19:18.905380 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 30153
2026-09-26T07:19:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2362 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30685](info): set up 2 sockets
Sep 26 07:19:18 dist-test-slave-7zlw krb5kdc[30685](info): commencing operation
krb5kdc: starting...
W20260926 07:19:19.976752 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:20 dist-test-slave-7zlw krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:20.104861 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:36339
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:36339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:20.218004 30698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:20.218356 30698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:20.218429 30698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:20.222589 30698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:20.222688 30698 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:19:20.222715 30698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:20.222748 30698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:20.222801 30698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:20.222838 30698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:20.228376 30698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:36339
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:36339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:20.229597 30698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:20.230755 30698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:19:20.239940 30698 server_base.cc:1034] running on GCE node
W20260926 07:19:20.239893 30706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:20.239900 30703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:20.239898 30704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:20.240803 30698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:20.242043 30698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:20.243247 30698 hybrid_clock.cc:648] HybridClock initialized: now 1790407160243229 us; error 32 us; skew 500 ppm
Sep 26 07:19:20 dist-test-slave-7zlw krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407160, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:20.253687 30698 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:20.254776 30698 webserver.cc:460] Webserver started at http://127.28.235.254:40067/ using document root <none> and password file <none>
I20260926 07:19:20.255462 30698 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:20.255568 30698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:20.255829 30698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:20.258260 30698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "c6f844423d184a71b9bda7e1d6da975c"
format_stamp: "Formatted at 2026-09-26 07:19:20 on dist-test-slave-7zlw"
server_key: "a7aeae6c55ef14970f631584fa59e661"
server_key_iv: "27c6e1feac1e64675b12c120251e7974"
server_key_version: "encryptionkey@0"
I20260926 07:19:20.258896 30698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "c6f844423d184a71b9bda7e1d6da975c"
format_stamp: "Formatted at 2026-09-26 07:19:20 on dist-test-slave-7zlw"
server_key: "a7aeae6c55ef14970f631584fa59e661"
server_key_iv: "27c6e1feac1e64675b12c120251e7974"
server_key_version: "encryptionkey@0"
I20260926 07:19:20.263144 30698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:19:20.266103 30713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:20.267544 30698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:20.267710 30698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "c6f844423d184a71b9bda7e1d6da975c"
format_stamp: "Formatted at 2026-09-26 07:19:20 on dist-test-slave-7zlw"
server_key: "a7aeae6c55ef14970f631584fa59e661"
server_key_iv: "27c6e1feac1e64675b12c120251e7974"
server_key_version: "encryptionkey@0"
I20260926 07:19:20.267841 30698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:20.283250 30698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:20.286659 30698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:20.286873 30698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:20.295589 30698 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:36339
I20260926 07:19:20.295608 30775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:36339 every 8 connection(s)
I20260926 07:19:20.296633 30698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:20.299656 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:20.301898 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 30698
I20260926 07:19:20.302001 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:20.302425 29615 external_mini_cluster.cc:1428] Setting key 8d8484467fc53ebd25493faed073cc4b
I20260926 07:19:20.306689 30776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c: Bootstrap starting.
I20260926 07:19:20.309299 30776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:20.310107 30776 log.cc:824] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:19:20 dist-test-slave-7zlw krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:20.312636 30776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c: No bootstrap required, opened a new log
I20260926 07:19:20.317332 30776 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f844423d184a71b9bda7e1d6da975c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36339 } }
I20260926 07:19:20.317600 30776 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:20.317662 30776 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f844423d184a71b9bda7e1d6da975c, State: Initialized, Role: FOLLOWER
I20260926 07:19:20.318157 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:20.304070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.28.235.254:36339)
0926 07:19:20.304574 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:20.304592 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:20.304634 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:19:20.305650 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:20.305673 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:20.306034 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:20.306454 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:20.307288 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.308261 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.309163 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.309697 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.309880 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:20.312707 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:20.312715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:20.312734 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:20.312774 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:20.314773 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:20.315347 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:20.315352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:20.315358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:20.315480 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:20.315802 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:20.315804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:20.315805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:20.316250 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:20.316430 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:20.316791 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 07:19:20.317197 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":145,"threads_started":1}
I20260926 07:19:20.318176 30776 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f844423d184a71b9bda7e1d6da975c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36339 } }
I20260926 07:19:20.318382 30776 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:20.318486 30776 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:20.318603 30776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:20.319792 30776 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f844423d184a71b9bda7e1d6da975c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36339 } }
I20260926 07:19:20.320187 30776 leader_election.cc:302] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6f844423d184a71b9bda7e1d6da975c; no voters:
I20260926 07:19:20.320575 30776 leader_election.cc:288] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:20.320750 30781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:20.320987 30781 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [term 1 LEADER]: Becoming Leader. State: Replica: c6f844423d184a71b9bda7e1d6da975c, State: Running, Role: LEADER
I20260926 07:19:20.321410 30781 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f844423d184a71b9bda7e1d6da975c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36339 } }
I20260926 07:19:20.321978 30776 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:20.323225 30782 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6f844423d184a71b9bda7e1d6da975c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f844423d184a71b9bda7e1d6da975c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36339 } } }
I20260926 07:19:20.323225 30783 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f844423d184a71b9bda7e1d6da975c. Latest consensus state: current_term: 1 leader_uuid: "c6f844423d184a71b9bda7e1d6da975c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f844423d184a71b9bda7e1d6da975c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36339 } } }
I20260926 07:19:20.323414 30782 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:20.323421 30783 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:20.323889 30790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:20.327461 30790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:20.333726 30790 catalog_manager.cc:1347] Generated new cluster ID: 202995575b744d2c8291e399e652a2f5
I20260926 07:19:20.333827 30790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:20.361169 30790 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:20.362246 30790 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:20.376801 30790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c: Generated new TSK 0
I20260926 07:19:20.377663 30790 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:19:20.408658 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:36339
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:20.523345 30802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:20.523625 30802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:20.523670 30802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:20.528980 30802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:20.529098 30802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:20.529145 30802 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:19:20.529189 30802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:20.535172 30802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.235.254:36339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:20.536284 30802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:20.537418 30802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:20.547653 30807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:20.547677 30808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:20.547678 30810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:20.547936 30802 server_base.cc:1034] running on GCE node
I20260926 07:19:20.548434 30802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:20.549175 30802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:20.550362 30802 hybrid_clock.cc:648] HybridClock initialized: now 1790407160550330 us; error 47 us; skew 500 ppm
Sep 26 07:19:20 dist-test-slave-7zlw krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407160, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:20.553215 30802 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:19:20.554039 30802 webserver.cc:460] Webserver started at http://127.28.235.193:45457/ using document root <none> and password file <none>
I20260926 07:19:20.554584 30802 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:20.554641 30802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:20.554805 30802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:20.556793 30802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "fd81a3b82295461295e57eda01374b48"
format_stamp: "Formatted at 2026-09-26 07:19:20 on dist-test-slave-7zlw"
server_key: "d83db7be145c62f3fdc24a4be3652098"
server_key_iv: "b5db8a628a57cd03c7ceb9136aa7e5ea"
server_key_version: "encryptionkey@0"
I20260926 07:19:20.557369 30802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "fd81a3b82295461295e57eda01374b48"
format_stamp: "Formatted at 2026-09-26 07:19:20 on dist-test-slave-7zlw"
server_key: "d83db7be145c62f3fdc24a4be3652098"
server_key_iv: "b5db8a628a57cd03c7ceb9136aa7e5ea"
server_key_version: "encryptionkey@0"
I20260926 07:19:20.560962 30802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:20.563373 30817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:20.564675 30802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:19:20.564919 30802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "fd81a3b82295461295e57eda01374b48"
format_stamp: "Formatted at 2026-09-26 07:19:20 on dist-test-slave-7zlw"
server_key: "d83db7be145c62f3fdc24a4be3652098"
server_key_iv: "b5db8a628a57cd03c7ceb9136aa7e5ea"
server_key_version: "encryptionkey@0"
I20260926 07:19:20.565078 30802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:20.590919 30802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:20.594416 30802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:20.594623 30802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:20.596129 30802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:20.596202 30802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:20.596287 30802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:20.596325 30802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:19:20 dist-test-slave-7zlw krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407160, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:20.607824 30802 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:40227
I20260926 07:19:20.607888 30932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:40227 every 8 connection(s)
I20260926 07:19:20.608711 30802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:20.610325 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:20.597239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:47547 (local address 127.28.235.254:36339)
0926 07:19:20.597366 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:20.597370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:20.598534 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 07:19:20.599748 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:20.599752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:20.599815 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:20.599902 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:20.601418 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.602050 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.603199 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.603508 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.603624 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:20.606499 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:20.606503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:20.606506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:20.606538 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:20.608143 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:20.608652 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:20.608656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:20.608658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:20.608743 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:20.609118 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:20.609121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:20.609123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:20.609358 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:20.609453 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:20.610067 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 07:19:20.610176 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:19:20.611066 30826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:20.597835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:36339 (local address 127.28.235.193:47547)
0926 07:19:20.598363 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:20.598408 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:20.599476 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:20.600095 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:20.600106 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:20.600568 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:20.601262 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:20.601279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.602177 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.602180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:20.603046 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:20.603079 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.603671 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.603675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:20.603860 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:20.604486 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:20.604502 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:20.606331 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:20.608261 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:20.608267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:20.608283 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:20.608547 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:20.608860 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:20.608863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:20.608870 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:20.608995 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:20.609506 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:20.609512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:20.609822 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 07:19:20.609979 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 07:19:20.610232 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":146,"threads_started":1}
I20260926 07:19:20.612424 30933 heartbeater.cc:347] Connected to a master server at 127.28.235.254:36339
I20260926 07:19:20.612579 30933 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:20.613130 30933 heartbeater.cc:511] Master 127.28.235.254:36339 requested a full tablet report, sending...
I20260926 07:19:20.614742 30729 ts_manager.cc:194] Registered new tserver with Master: fd81a3b82295461295e57eda01374b48 (127.28.235.193:40227)
I20260926 07:19:20.615442 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 30802
I20260926 07:19:20.615561 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:20.615872 29615 external_mini_cluster.cc:1428] Setting key f2179d943e7648d9d7e86061c94f0ab2
I20260926 07:19:20.616328 30729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:47547
I20260926 07:19:20.622390 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:20.614558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:35181 (local address 127.28.235.254:36339)
0926 07:19:20.614684 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:20.614687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:20.614820 (+ 133us) server_negotiation.cc:407] Connection header received
0926 07:19:20.614948 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:20.614951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:20.614997 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:20.615055 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:20.615566 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.616337 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.617443 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.617648 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.617735 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:20.618988 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:20.618994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:20.619001 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:20.619032 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:20.620556 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:20.621111 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:20.621114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:20.621115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:20.621187 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:20.621525 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:20.621532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:20.621533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:20.621696 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:20.621847 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:20.622114 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 07:19:20.622222 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:19:20.622409 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:20.614430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:36339 (local address 127.28.235.193:35181)
0926 07:19:20.614725 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:20.614749 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:20.614853 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:20.615098 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:20.615101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:20.615311 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:20.615430 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:20.615438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.616512 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.616515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:20.617338 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:20.617349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.617767 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.617771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:20.617878 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:20.618402 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:20.618416 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:20.618841 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:20.620669 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:20.620673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:20.620677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:20.620964 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:20.621283 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:20.621289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:20.621294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:20.621398 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:20.621848 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:20.621852 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:20.621945 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 07:19:20.622049 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 07:19:20.622198 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:19:20.644830 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:36339
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:20.757375 30942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:20.757633 30942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:20.757679 30942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:20.761705 30942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:20.761762 30942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:20.761801 30942 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:19:20.761834 30942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:20.767123 30942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.235.254:36339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:20.768167 30942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:20.769241 30942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:20.779086 30947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:20.779099 30948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:20.779095 30950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:20.779277 30942 server_base.cc:1034] running on GCE node
I20260926 07:19:20.779762 30942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:20.780417 30942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:20.781570 30942 hybrid_clock.cc:648] HybridClock initialized: now 1790407160781555 us; error 27 us; skew 500 ppm
Sep 26 07:19:20 dist-test-slave-7zlw krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407160, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:20.784238 30942 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:19:20.785143 30942 webserver.cc:460] Webserver started at http://127.28.235.194:38319/ using document root <none> and password file <none>
I20260926 07:19:20.785662 30942 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:20.785720 30942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:20.785889 30942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:20.788010 30942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "3959e6d4a6ee42309fc984ec57426013"
format_stamp: "Formatted at 2026-09-26 07:19:20 on dist-test-slave-7zlw"
server_key: "74e0269ce79863563a40719400a97ffe"
server_key_iv: "9b3ab6a4a0b21ab2c8c3fd45814a506f"
server_key_version: "encryptionkey@0"
I20260926 07:19:20.788501 30942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "3959e6d4a6ee42309fc984ec57426013"
format_stamp: "Formatted at 2026-09-26 07:19:20 on dist-test-slave-7zlw"
server_key: "74e0269ce79863563a40719400a97ffe"
server_key_iv: "9b3ab6a4a0b21ab2c8c3fd45814a506f"
server_key_version: "encryptionkey@0"
I20260926 07:19:20.792268 30942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 07:19:20.794631 30957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:20.795832 30942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:19:20.795962 30942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "3959e6d4a6ee42309fc984ec57426013"
format_stamp: "Formatted at 2026-09-26 07:19:20 on dist-test-slave-7zlw"
server_key: "74e0269ce79863563a40719400a97ffe"
server_key_iv: "9b3ab6a4a0b21ab2c8c3fd45814a506f"
server_key_version: "encryptionkey@0"
I20260926 07:19:20.796070 30942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:20.820647 30942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:20.824083 30942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:20.824280 30942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:20.825856 30942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:20.825927 30942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:20.826022 30942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:20.826069 30942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:19:20 dist-test-slave-7zlw krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407160, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:20.837404 30942 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:45301
I20260926 07:19:20.837472 31072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:45301 every 8 connection(s)
I20260926 07:19:20.838335 30942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:20.839785 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:20.826531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:58847 (local address 127.28.235.254:36339)
0926 07:19:20.826689 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:20.826696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:20.828057 (+ 1361us) server_negotiation.cc:407] Connection header received
0926 07:19:20.829272 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:20.829276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:20.829334 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:20.829408 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:20.830756 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.831366 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.832508 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.832733 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.832841 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:20.835974 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:20.835978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:20.835980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:20.836008 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:20.837677 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:20.838220 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:20.838226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:20.838230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:20.838319 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:20.838662 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:20.838667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:20.838669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:20.838923 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:20.839029 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:20.839542 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 07:19:20.839645 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:19:20.840540 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:20.827429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:36339 (local address 127.28.235.194:58847)
0926 07:19:20.827890 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:20.827942 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:20.829009 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:20.829585 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:20.829592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:20.829998 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:20.830605 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:20.830619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.831486 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.831489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:20.832378 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:20.832388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.832893 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.832897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:20.833134 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:20.833898 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:20.833917 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:20.835814 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:20.837794 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:20.837800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:20.837816 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:20.838118 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:20.838406 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:20.838411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:20.838414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:20.838571 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:20.839082 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:20.839089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:20.839319 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 07:19:20.839472 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 07:19:20.839721 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":133,"threads_started":1}
I20260926 07:19:20.841305 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 30942
I20260926 07:19:20.841409 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:20.841684 29615 external_mini_cluster.cc:1428] Setting key 5eca0cb6cdb2497c106a5bbe2a8355d4
I20260926 07:19:20.841876 31073 heartbeater.cc:347] Connected to a master server at 127.28.235.254:36339
I20260926 07:19:20.842015 31073 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:20.842585 31073 heartbeater.cc:511] Master 127.28.235.254:36339 requested a full tablet report, sending...
I20260926 07:19:20.843683 30730 ts_manager.cc:194] Registered new tserver with Master: 3959e6d4a6ee42309fc984ec57426013 (127.28.235.194:45301)
I20260926 07:19:20.844496 30730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:58847
I20260926 07:19:20.852444 31079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:20.844079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:36339 (local address 127.28.235.194:46453)
0926 07:19:20.844386 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:20.844398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:20.844504 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:20.844838 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:20.844841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:20.845087 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:20.845339 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:20.845349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.846335 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.846338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:20.846926 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:20.846935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.847576 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.847579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:20.847648 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:20.848207 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:20.848218 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:20.848553 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:20.850686 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:20.850689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:20.850690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:20.850923 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:20.851304 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:20.851306 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:20.851308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:20.851398 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:20.851915 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:20.851917 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:20.852051 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 07:19:20.852149 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:19:20.852268 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":5,"thread_start_us":88,"threads_started":1}
I20260926 07:19:20.852558 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:20.844128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:46453 (local address 127.28.235.254:36339)
0926 07:19:20.844239 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:20.844243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:20.844465 (+ 222us) server_negotiation.cc:407] Connection header received
0926 07:19:20.844580 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:20.844583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:20.844635 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:20.844714 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:20.845473 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.846192 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.847042 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:20.847450 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:20.847564 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:20.848692 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:20.848699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:20.848705 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:20.848737 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:20.850563 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:20.851083 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:20.851089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:20.851092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:20.851186 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:20.851499 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:20.851502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:20.851504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:20.851805 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:20.851914 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:20.852255 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 07:19:20.852387 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:19:20.873248 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:36339
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:20.986935 31082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:20.987217 31082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:20.987303 31082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:20.991369 31082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:20.991428 31082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:20.991505 31082 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:19:20.991591 31082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:20.997877 31082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.235.254:36339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:20.999119 31082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:21.000267 31082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:21.011386 31087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:21.011386 31088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:21.011421 31090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:21.011381 31082 server_base.cc:1034] running on GCE node
I20260926 07:19:21.012112 31082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:21.012792 31082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:21.013975 31082 hybrid_clock.cc:648] HybridClock initialized: now 1790407161013961 us; error 68 us; skew 500 ppm
Sep 26 07:19:21 dist-test-slave-7zlw krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407161, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:21.016666 31082 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:19:21.017599 31082 webserver.cc:460] Webserver started at http://127.28.235.195:39943/ using document root <none> and password file <none>
I20260926 07:19:21.018169 31082 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:21.018278 31082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:21.018512 31082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:21.020591 31082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "338480c296d447f3b3c6c425c742149f"
format_stamp: "Formatted at 2026-09-26 07:19:21 on dist-test-slave-7zlw"
server_key: "d681bce470e1f51e1d75a36d375ee7b0"
server_key_iv: "046b93174735ec93011d4af4532fea1b"
server_key_version: "encryptionkey@0"
I20260926 07:19:21.021263 31082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "338480c296d447f3b3c6c425c742149f"
format_stamp: "Formatted at 2026-09-26 07:19:21 on dist-test-slave-7zlw"
server_key: "d681bce470e1f51e1d75a36d375ee7b0"
server_key_iv: "046b93174735ec93011d4af4532fea1b"
server_key_version: "encryptionkey@0"
I20260926 07:19:21.025422 31082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:19:21.027987 31097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:21.029589 31082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:21.029721 31082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "338480c296d447f3b3c6c425c742149f"
format_stamp: "Formatted at 2026-09-26 07:19:21 on dist-test-slave-7zlw"
server_key: "d681bce470e1f51e1d75a36d375ee7b0"
server_key_iv: "046b93174735ec93011d4af4532fea1b"
server_key_version: "encryptionkey@0"
I20260926 07:19:21.029850 31082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:21.039754 31082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:21.042930 31082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:21.043119 31082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:21.044538 31082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:21.044608 31082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:21.044677 31082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:21.044721 31082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:19:21 dist-test-slave-7zlw krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407161, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:21.055943 31082 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:39559
I20260926 07:19:21.056013 31212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:39559 every 8 connection(s)
I20260926 07:19:21.057004 31082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:21.059754 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.045574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:38763 (local address 127.28.235.254:36339)
0926 07:19:21.045704 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.045708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.046804 (+ 1096us) server_negotiation.cc:407] Connection header received
0926 07:19:21.047972 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.047975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.048030 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.048187 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:21.049966 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.050584 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.052551 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.052842 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.052929 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.056196 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:21.056200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:21.056202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:21.056235 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:21.057841 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.058367 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.058370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.058372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.058451 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.058711 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.058714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.058715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.058891 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:21.058967 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.059496 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.059626 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:19:21.059827 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 31082
I20260926 07:19:21.059944 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:21.060215 29615 external_mini_cluster.cc:1428] Setting key fcab96ce5acbdf34375f89471d74cd9a
I20260926 07:19:21.060777 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.046117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:36339 (local address 127.28.235.195:38763)
0926 07:19:21.046651 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.046693 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.047747 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.048406 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.048420 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.048934 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:21.049766 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.049792 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.050750 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.050757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.052388 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.052405 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.053007 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.053037 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.053271 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.054053 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:21.054079 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:21.055977 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:21.057948 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:21.057953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:21.057967 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:21.058273 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:21.058514 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:21.058516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:21.058517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:21.058638 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:21.058980 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:21.058985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:21.059281 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.059441 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.059772 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":142,"threads_started":1}
I20260926 07:19:21.062023 31213 heartbeater.cc:347] Connected to a master server at 127.28.235.254:36339
I20260926 07:19:21.062296 31213 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:21.062834 31213 heartbeater.cc:511] Master 127.28.235.254:36339 requested a full tablet report, sending...
I20260926 07:19:21.063898 30729 ts_manager.cc:194] Registered new tserver with Master: 338480c296d447f3b3c6c425c742149f (127.28.235.195:39559)
I20260926 07:19:21.064661 30729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:38763
I20260926 07:19:21.071871 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.064395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:59559 (local address 127.28.235.254:36339)
0926 07:19:21.064520 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.064524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.064809 (+ 285us) server_negotiation.cc:407] Connection header received
0926 07:19:21.064887 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.064905 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.064965 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.065075 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:21.065606 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.066387 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.067293 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.067511 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.067598 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.068874 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:21.068878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:21.068881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:21.068905 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:21.070236 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.070649 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.070653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.070655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.070741 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.071034 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.071040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.071041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.071227 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:21.071333 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.071598 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.071723 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:19:21.071882 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.064290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:36339 (local address 127.28.235.195:59559)
0926 07:19:21.064672 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.064698 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.064799 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.065091 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.065094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.065371 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:21.065468 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.065474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.066546 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.066549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.067166 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.067174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.067638 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.067641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.067715 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.068301 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:21.068314 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:21.068753 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:21.070347 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:21.070352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:21.070354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:21.070550 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:21.070835 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:21.070840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:21.070842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:21.070927 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:21.071333 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:21.071337 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:21.071433 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.071535 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.071684 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":101,"threads_started":1}
I20260926 07:19:21.075924 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:19:21 dist-test-slave-7zlw krb5kdc[30685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:19:21 dist-test-slave-7zlw krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:21.102458 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.094044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.28.235.254:36339)
0926 07:19:21.094169 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.094173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.094187 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:19:21.094246 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.094248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.094290 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.094368 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:21.095051 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.095601 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.096404 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.096662 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.096757 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.099249 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:21.099253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:21.099257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:21.099293 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:21.100775 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.101272 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.101275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.101276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.101378 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.101680 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.101682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.101683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.101875 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:21.101990 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.102230 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.102329 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:19:21.105582 30729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:19:21.108678 30729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:21.123961 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.117667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.28.235.193:40227)
0926 07:19:21.118856 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.118864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.118887 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:19:21.118967 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.118972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.119148 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.119323 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:21.119664 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.120826 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.122448 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.123178 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.123311 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.123451 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.123715 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.123843 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1070,"thread_start_us":97,"threads_started":1}
I20260926 07:19:21.124039 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.118678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.28.235.194:45301)
0926 07:19:21.119156 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.119167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.119186 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:19:21.119260 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.119264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.119402 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.119554 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:21.120011 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.121115 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.122485 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.123322 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.123552 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.123744 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.123797 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.123887 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":118,"threads_started":1}
I20260926 07:19:21.124445 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.117493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:40227 (local address 127.0.0.1:53634)
0926 07:19:21.117856 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.117914 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.118062 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.119315 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.119322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.119344 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:21.119525 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.119534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.120943 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.120946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.122278 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.122285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.123296 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.123300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.123409 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.123446 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.123557 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.124235 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":8,"thread_start_us":87,"threads_started":1}
I20260926 07:19:21.124534 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.118327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.28.235.195:39559)
0926 07:19:21.119077 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.119090 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.119114 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:19:21.119194 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.119200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.119398 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.119555 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:21.119780 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.120597 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.122395 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.123322 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.123826 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.123938 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.123982 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.124068 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":133,"threads_started":1}
I20260926 07:19:21.124847 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.118228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:45301 (local address 127.0.0.1:48680)
0926 07:19:21.118644 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.118655 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.118734 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.119738 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.119744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.119764 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:21.119881 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.119888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.121256 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.121260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.122316 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.122346 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.123468 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.123472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.123628 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.123649 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.123718 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.124414 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":691,"thread_start_us":87,"threads_started":1}
I20260926 07:19:21.124464 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.117854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:39559 (local address 127.0.0.1:56616)
0926 07:19:21.118408 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.118424 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.118515 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.119539 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.119542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.119561 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:21.119666 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.119673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.120722 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.120726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.122261 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.122272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.123468 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.123472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.123584 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.123606 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.123680 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.124250 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":474,"thread_start_us":114,"threads_started":1}
I20260926 07:19:21.127571 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 281356dec0e7443c9885c777c0f44de2 (DEFAULT_TABLE table=test-table [id=36e7b85e4b72462d8d4d6af469604170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:21.127645 30867 tablet_service.cc:1467] Processing CreateTablet for tablet 281356dec0e7443c9885c777c0f44de2 (DEFAULT_TABLE table=test-table [id=36e7b85e4b72462d8d4d6af469604170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:21.128309 31147 tablet_service.cc:1467] Processing CreateTablet for tablet 281356dec0e7443c9885c777c0f44de2 (DEFAULT_TABLE table=test-table [id=36e7b85e4b72462d8d4d6af469604170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:21.128706 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281356dec0e7443c9885c777c0f44de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:21.128866 30867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281356dec0e7443c9885c777c0f44de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:21.129282 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281356dec0e7443c9885c777c0f44de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:21.135576 31236 tablet_bootstrap.cc:492] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: Bootstrap starting.
I20260926 07:19:21.136817 31237 tablet_bootstrap.cc:492] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: Bootstrap starting.
I20260926 07:19:21.137346 31238 tablet_bootstrap.cc:492] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: Bootstrap starting.
I20260926 07:19:21.138088 31236 tablet_bootstrap.cc:654] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:21.138815 31236 log.cc:824] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:21.138957 31237 tablet_bootstrap.cc:654] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:21.139252 31238 tablet_bootstrap.cc:654] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:21.139804 31237 log.cc:824] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:21.140010 31238 log.cc:824] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:21.141352 31236 tablet_bootstrap.cc:492] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: No bootstrap required, opened a new log
I20260926 07:19:21.141546 31236 ts_tablet_manager.cc:1397] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260926 07:19:21.141654 31237 tablet_bootstrap.cc:492] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: No bootstrap required, opened a new log
I20260926 07:19:21.141834 31237 ts_tablet_manager.cc:1397] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260926 07:19:21.141996 31238 tablet_bootstrap.cc:492] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: No bootstrap required, opened a new log
I20260926 07:19:21.142166 31238 ts_tablet_manager.cc:1397] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:19:21.145313 31237 raft_consensus.cc:348] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.145550 31237 raft_consensus.cc:374] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:21.145643 31237 raft_consensus.cc:729] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd81a3b82295461295e57eda01374b48, State: Initialized, Role: FOLLOWER
I20260926 07:19:21.145936 31238 raft_consensus.cc:348] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.146097 31237 consensus_queue.cc:260] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.146186 31238 raft_consensus.cc:374] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:21.145569 31236 raft_consensus.cc:348] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.146358 31236 raft_consensus.cc:374] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:21.146417 31236 raft_consensus.cc:729] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3959e6d4a6ee42309fc984ec57426013, State: Initialized, Role: FOLLOWER
I20260926 07:19:21.146767 31238 raft_consensus.cc:729] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338480c296d447f3b3c6c425c742149f, State: Initialized, Role: FOLLOWER
I20260926 07:19:21.146836 30933 heartbeater.cc:503] Master 127.28.235.254:36339 was elected leader, sending a full tablet report...
I20260926 07:19:21.146859 31236 consensus_queue.cc:260] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.147111 31237 ts_tablet_manager.cc:1428] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260926 07:19:21.147246 31238 consensus_queue.cc:260] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.147904 31213 heartbeater.cc:503] Master 127.28.235.254:36339 was elected leader, sending a full tablet report...
I20260926 07:19:21.148115 31238 ts_tablet_manager.cc:1428] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:19:21.148617 31236 ts_tablet_manager.cc:1428] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260926 07:19:21.148681 31073 heartbeater.cc:503] Master 127.28.235.254:36339 was elected leader, sending a full tablet report...
I20260926 07:19:21.227701 31242 raft_consensus.cc:482] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:21.227983 31242 raft_consensus.cc:504] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.229290 31242 leader_election.cc:288] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3959e6d4a6ee42309fc984ec57426013 (127.28.235.194:45301), 338480c296d447f3b3c6c425c742149f (127.28.235.195:39559)
I20260926 07:19:21.235076 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.229879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:51833 (local address 127.28.235.195:39559)
0926 07:19:21.230014 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.230017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.230262 (+ 245us) server_negotiation.cc:407] Connection header received
0926 07:19:21.230379 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.230382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.230436 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.230523 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:21.230949 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.231769 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.233689 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.234360 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.234487 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.234542 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.234850 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.234932 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:19:21.235075 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.229676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:39395 (local address 127.28.235.194:45301)
0926 07:19:21.229872 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.229878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.230202 (+ 324us) server_negotiation.cc:407] Connection header received
0926 07:19:21.230380 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.230383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.230436 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.230523 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:21.230949 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.231768 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.233689 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.234360 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.234487 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.234542 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.234850 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.234932 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 07:19:21.235167 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.229566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:45301 (local address 127.28.235.193:39395)
0926 07:19:21.230061 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.230076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.230262 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.230549 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.230552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.230594 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:21.230832 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.230839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.231886 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.231890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.233546 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.233560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.234489 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.234493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.234663 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.234737 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.234818 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.234939 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":262,"spinlock_wait_cycles":7808,"thread_start_us":149,"threads_started":1}
I20260926 07:19:21.235165 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.229662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:39559 (local address 127.28.235.193:51833)
0926 07:19:21.230150 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.230182 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.230269 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.230549 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.230552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.230593 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:21.230832 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.230839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.231886 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.231890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.233546 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.233560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.234489 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.234493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.234703 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.234737 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.234818 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.234942 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":312,"spinlock_wait_cycles":28672,"thread_start_us":127,"threads_started":1}
I20260926 07:19:21.235841 31167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281356dec0e7443c9885c777c0f44de2" candidate_uuid: "fd81a3b82295461295e57eda01374b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338480c296d447f3b3c6c425c742149f" is_pre_election: true
I20260926 07:19:21.235842 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281356dec0e7443c9885c777c0f44de2" candidate_uuid: "fd81a3b82295461295e57eda01374b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3959e6d4a6ee42309fc984ec57426013" is_pre_election: true
I20260926 07:19:21.236162 31167 raft_consensus.cc:2454] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd81a3b82295461295e57eda01374b48 in term 0.
I20260926 07:19:21.236169 31027 raft_consensus.cc:2454] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd81a3b82295461295e57eda01374b48 in term 0.
I20260926 07:19:21.236670 30821 leader_election.cc:302] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338480c296d447f3b3c6c425c742149f, fd81a3b82295461295e57eda01374b48; no voters:
I20260926 07:19:21.237073 31242 raft_consensus.cc:2781] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:21.237164 31242 raft_consensus.cc:482] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:21.237215 31242 raft_consensus.cc:3037] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:21.238561 31242 raft_consensus.cc:504] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.238986 31242 leader_election.cc:288] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [CANDIDATE]: Term 1 election: Requested vote from peers 3959e6d4a6ee42309fc984ec57426013 (127.28.235.194:45301), 338480c296d447f3b3c6c425c742149f (127.28.235.195:39559)
I20260926 07:19:21.239382 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281356dec0e7443c9885c777c0f44de2" candidate_uuid: "fd81a3b82295461295e57eda01374b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3959e6d4a6ee42309fc984ec57426013"
I20260926 07:19:21.239409 31167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281356dec0e7443c9885c777c0f44de2" candidate_uuid: "fd81a3b82295461295e57eda01374b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "338480c296d447f3b3c6c425c742149f"
I20260926 07:19:21.239506 31027 raft_consensus.cc:3037] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:21.239562 31167 raft_consensus.cc:3037] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:21.240887 31167 raft_consensus.cc:2454] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd81a3b82295461295e57eda01374b48 in term 1.
I20260926 07:19:21.240893 31027 raft_consensus.cc:2454] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd81a3b82295461295e57eda01374b48 in term 1.
I20260926 07:19:21.241271 30821 leader_election.cc:302] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338480c296d447f3b3c6c425c742149f, fd81a3b82295461295e57eda01374b48; no voters:
I20260926 07:19:21.241473 31242 raft_consensus.cc:2781] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:21.241744 31242 raft_consensus.cc:686] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 1 LEADER]: Becoming Leader. State: Replica: fd81a3b82295461295e57eda01374b48, State: Running, Role: LEADER
I20260926 07:19:21.242164 31242 consensus_queue.cc:237] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.245546 30728 catalog_manager.cc:5495] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd81a3b82295461295e57eda01374b48 (127.28.235.193). New cstate: current_term: 1 leader_uuid: "fd81a3b82295461295e57eda01374b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:21.282351 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.278534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.28.235.193:40227)
0926 07:19:21.278686 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.278692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.278707 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:19:21.278932 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.278935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.278991 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.279093 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:21.279360 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.279865 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.280770 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.280985 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.281145 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.281493 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:21.281926 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:21.282054 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.282132 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.282203 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:19:21.288566 31167 raft_consensus.cc:1261] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f [term 1 FOLLOWER]: Refusing update from remote peer fd81a3b82295461295e57eda01374b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:21.288566 31027 raft_consensus.cc:1261] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013 [term 1 FOLLOWER]: Refusing update from remote peer fd81a3b82295461295e57eda01374b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:21.289291 31247 consensus_queue.cc:1035] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:21.289475 31242 consensus_queue.cc:1035] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:21.299990 31250 mvcc.cc:205] Tried to move back new op lower bound from 7333507732629938176 to 7333507732451012608. Current Snapshot: MvccSnapshot[applied={T|T < 7333507732629938176}]
I20260926 07:19:21.306901 31251 mvcc.cc:205] Tried to move back new op lower bound from 7333507732629938176 to 7333507732451012608. Current Snapshot: MvccSnapshot[applied={T|T < 7333507732629938176 or (T in {7333507732629938176})}]
W20260926 07:19:21.308175 31214 tablet.cc:2369] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:19:21.309206 31252 mvcc.cc:205] Tried to move back new op lower bound from 7333507732629938176 to 7333507732451012608. Current Snapshot: MvccSnapshot[applied={T|T < 7333507732629938176}]
I20260926 07:19:21.314172 30728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:21.314345 30728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:21.316973 30728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c: Sending DeleteTablet for 3 replicas of tablet 281356dec0e7443c9885c777c0f44de2
I20260926 07:19:21.317883 31147 tablet_service.cc:1514] Processing DeleteTablet for tablet 281356dec0e7443c9885c777c0f44de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:21 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:56616
I20260926 07:19:21.317919 30867 tablet_service.cc:1514] Processing DeleteTablet for tablet 281356dec0e7443c9885c777c0f44de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:21 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:53634
I20260926 07:19:21.318435 31260 tablet_replica.cc:330] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: stopping tablet replica
I20260926 07:19:21.318471 31007 tablet_service.cc:1514] Processing DeleteTablet for tablet 281356dec0e7443c9885c777c0f44de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:21 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:48680
I20260926 07:19:21.318910 31260 raft_consensus.cc:2229] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:21.318955 31261 tablet_replica.cc:330] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: stopping tablet replica
I20260926 07:19:21.319234 31261 raft_consensus.cc:2229] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:21.319468 31260 raft_consensus.cc:2258] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:21.319401 30728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:19:21.319518 31261 raft_consensus.cc:2258] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:21.319842 31262 tablet_replica.cc:330] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: stopping tablet replica
W20260926 07:19:21.319849 30728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:21.320138 31262 raft_consensus.cc:2229] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:21.320433 31262 raft_consensus.cc:2258] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:21.321905 31261 ts_tablet_manager.cc:1905] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:21.322314 31260 ts_tablet_manager.cc:1905] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:21.324378 31262 ts_tablet_manager.cc:1905] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:21.325076 31261 ts_tablet_manager.cc:1918] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:21.325220 31261 log.cc:1196] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/281356dec0e7443c9885c777c0f44de2
I20260926 07:19:21.325460 31147 tablet_service.cc:1467] Processing CreateTablet for tablet 3548591940a9425ca44e55452ce9074f (DEFAULT_TABLE table=test-table [id=2a78d324358146a796aa44e2a748013f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:21.325724 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3548591940a9425ca44e55452ce9074f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:21.325672 30867 tablet_service.cc:1467] Processing CreateTablet for tablet 3548591940a9425ca44e55452ce9074f (DEFAULT_TABLE table=test-table [id=2a78d324358146a796aa44e2a748013f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:21.325913 31261 ts_tablet_manager.cc:1939] T 281356dec0e7443c9885c777c0f44de2 P 338480c296d447f3b3c6c425c742149f: Deleting consensus metadata
I20260926 07:19:21.325949 30867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3548591940a9425ca44e55452ce9074f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:21.327796 31238 tablet_bootstrap.cc:492] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f: Bootstrap starting.
I20260926 07:19:21.328037 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 3548591940a9425ca44e55452ce9074f (DEFAULT_TABLE table=test-table [id=2a78d324358146a796aa44e2a748013f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:21.328202 31237 tablet_bootstrap.cc:492] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: Bootstrap starting.
I20260926 07:19:21.328305 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3548591940a9425ca44e55452ce9074f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:21.328774 31238 tablet_bootstrap.cc:654] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:21.329183 31237 tablet_bootstrap.cc:654] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:21.330480 31237 tablet_bootstrap.cc:492] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: No bootstrap required, opened a new log
I20260926 07:19:21.330577 31237 ts_tablet_manager.cc:1397] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:19:21.330873 31236 tablet_bootstrap.cc:492] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: Bootstrap starting.
I20260926 07:19:21.331009 31237 raft_consensus.cc:348] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.331128 31237 raft_consensus.cc:374] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:21.331180 31237 raft_consensus.cc:729] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd81a3b82295461295e57eda01374b48, State: Initialized, Role: FOLLOWER
I20260926 07:19:21.331297 31237 consensus_queue.cc:260] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.331552 31237 ts_tablet_manager.cc:1428] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:21.331839 31236 tablet_bootstrap.cc:654] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:21.332573 30717 catalog_manager.cc:4841] TS 338480c296d447f3b3c6c425c742149f (127.28.235.195:39559): tablet 281356dec0e7443c9885c777c0f44de2 (table test-table [id=36e7b85e4b72462d8d4d6af469604170]) successfully deleted
I20260926 07:19:21.333308 31236 tablet_bootstrap.cc:492] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: No bootstrap required, opened a new log
I20260926 07:19:21.333395 31236 ts_tablet_manager.cc:1397] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 07:19:21.333840 31236 raft_consensus.cc:348] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.333952 31236 raft_consensus.cc:374] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:21.333994 31236 raft_consensus.cc:729] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3959e6d4a6ee42309fc984ec57426013, State: Initialized, Role: FOLLOWER
I20260926 07:19:21.334134 31236 consensus_queue.cc:260] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.334412 31236 ts_tablet_manager.cc:1428] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:19:21.336162 31260 ts_tablet_manager.cc:1918] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:21.336277 31260 log.cc:1196] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/281356dec0e7443c9885c777c0f44de2
I20260926 07:19:21.336567 31260 ts_tablet_manager.cc:1939] T 281356dec0e7443c9885c777c0f44de2 P fd81a3b82295461295e57eda01374b48: Deleting consensus metadata
I20260926 07:19:21.338045 30716 catalog_manager.cc:4841] TS fd81a3b82295461295e57eda01374b48 (127.28.235.193:40227): tablet 281356dec0e7443c9885c777c0f44de2 (table test-table [id=36e7b85e4b72462d8d4d6af469604170]) successfully deleted
I20260926 07:19:21.338763 31262 ts_tablet_manager.cc:1918] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:21.338878 31262 log.cc:1196] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/281356dec0e7443c9885c777c0f44de2
I20260926 07:19:21.339161 31262 ts_tablet_manager.cc:1939] T 281356dec0e7443c9885c777c0f44de2 P 3959e6d4a6ee42309fc984ec57426013: Deleting consensus metadata
W20260926 07:19:21.339462 31074 tablet.cc:2369] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:19:21.340405 30716 catalog_manager.cc:4841] TS 3959e6d4a6ee42309fc984ec57426013 (127.28.235.194:45301): tablet 281356dec0e7443c9885c777c0f44de2 (table test-table [id=36e7b85e4b72462d8d4d6af469604170]) successfully deleted
I20260926 07:19:21.341217 31238 tablet_bootstrap.cc:492] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f: No bootstrap required, opened a new log
I20260926 07:19:21.341339 31238 ts_tablet_manager.cc:1397] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260926 07:19:21.341929 31238 raft_consensus.cc:348] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.342077 31238 raft_consensus.cc:374] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:21.342116 31238 raft_consensus.cc:729] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338480c296d447f3b3c6c425c742149f, State: Initialized, Role: FOLLOWER
I20260926 07:19:21.342300 31238 consensus_queue.cc:260] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.342603 31238 ts_tablet_manager.cc:1428] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260926 07:19:21.360102 30934 tablet.cc:2369] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:19:21.363732 31244 raft_consensus.cc:482] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:21.363878 31244 raft_consensus.cc:504] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.365157 31244 leader_election.cc:288] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd81a3b82295461295e57eda01374b48 (127.28.235.193:40227), 3959e6d4a6ee42309fc984ec57426013 (127.28.235.194:45301)
I20260926 07:19:21.370095 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.366043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:48799 (local address 127.28.235.194:45301)
0926 07:19:21.366174 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.366178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.366190 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:19:21.366254 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.366260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.366319 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.366390 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:21.366644 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.367128 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.368922 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.369446 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.369581 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.369637 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.369863 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.369941 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:19:21.370220 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.365409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:40227 (local address 127.28.235.195:52867)
0926 07:19:21.365539 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.365555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.365646 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.366026 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.366030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.366048 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:21.366194 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.366202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.367227 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.367231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.368825 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.368836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.369816 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.369820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.369925 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.369946 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.370021 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.370087 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":165,"spinlock_wait_cycles":1792}
I20260926 07:19:21.370246 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.365510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:52867 (local address 127.28.235.193:40227)
0926 07:19:21.365655 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.365659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.365676 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:19:21.365758 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.365764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.365822 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.365905 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:21.366322 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.367109 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.369029 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.369678 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.369809 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.369860 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.370048 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.370123 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:19:21.370545 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.365689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:45301 (local address 127.28.235.195:48799)
0926 07:19:21.365976 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.365989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.366096 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.366398 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.366401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.366416 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:21.366536 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.366541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.367269 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.367272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.368795 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.368811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.369581 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.369586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.369680 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.369701 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.370312 (+ 611us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.370396 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":77,"spinlock_wait_cycles":28800,"thread_start_us":94,"threads_started":1}
I20260926 07:19:21.370987 30887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3548591940a9425ca44e55452ce9074f" candidate_uuid: "338480c296d447f3b3c6c425c742149f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd81a3b82295461295e57eda01374b48" is_pre_election: true
I20260926 07:19:21.371246 30887 raft_consensus.cc:2454] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 338480c296d447f3b3c6c425c742149f in term 0.
I20260926 07:19:21.371694 31100 leader_election.cc:302] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338480c296d447f3b3c6c425c742149f, fd81a3b82295461295e57eda01374b48; no voters:
I20260926 07:19:21.371733 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3548591940a9425ca44e55452ce9074f" candidate_uuid: "338480c296d447f3b3c6c425c742149f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3959e6d4a6ee42309fc984ec57426013" is_pre_election: true
I20260926 07:19:21.371865 31027 raft_consensus.cc:2454] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 338480c296d447f3b3c6c425c742149f in term 0.
I20260926 07:19:21.372038 31244 raft_consensus.cc:2781] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:21.372104 31244 raft_consensus.cc:482] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:21.372143 31244 raft_consensus.cc:3037] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:21.373513 31244 raft_consensus.cc:504] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.373893 31244 leader_election.cc:288] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [CANDIDATE]: Term 1 election: Requested vote from peers fd81a3b82295461295e57eda01374b48 (127.28.235.193:40227), 3959e6d4a6ee42309fc984ec57426013 (127.28.235.194:45301)
I20260926 07:19:21.374344 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3548591940a9425ca44e55452ce9074f" candidate_uuid: "338480c296d447f3b3c6c425c742149f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3959e6d4a6ee42309fc984ec57426013"
I20260926 07:19:21.374480 31027 raft_consensus.cc:3037] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:21.374629 30887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3548591940a9425ca44e55452ce9074f" candidate_uuid: "338480c296d447f3b3c6c425c742149f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd81a3b82295461295e57eda01374b48"
I20260926 07:19:21.374744 30887 raft_consensus.cc:3037] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:21.375367 31027 raft_consensus.cc:2454] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 338480c296d447f3b3c6c425c742149f in term 1.
I20260926 07:19:21.375916 30887 raft_consensus.cc:2454] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 338480c296d447f3b3c6c425c742149f in term 1.
I20260926 07:19:21.376407 31100 leader_election.cc:302] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338480c296d447f3b3c6c425c742149f, 3959e6d4a6ee42309fc984ec57426013; no voters:
I20260926 07:19:21.376627 31244 raft_consensus.cc:2781] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:21.376840 31244 raft_consensus.cc:686] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 1 LEADER]: Becoming Leader. State: Replica: 338480c296d447f3b3c6c425c742149f, State: Running, Role: LEADER
I20260926 07:19:21.377123 31244 consensus_queue.cc:237] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.379685 30728 catalog_manager.cc:5495] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f reported cstate change: term changed from 0 to 1, leader changed from <none> to 338480c296d447f3b3c6c425c742149f (127.28.235.195). New cstate: current_term: 1 leader_uuid: "338480c296d447f3b3c6c425c742149f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } health_report { overall_health: HEALTHY } } }
Sep 26 07:19:21 dist-test-slave-7zlw krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407160, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:21.420759 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.410982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50164 (local address 127.28.235.254:36339)
0926 07:19:21.411136 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.411142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.411407 (+ 265us) server_negotiation.cc:407] Connection header received
0926 07:19:21.411564 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.411566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.411608 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.411678 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:21.412531 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.413084 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.414124 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.414351 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.414438 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.417130 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:21.417133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:21.417135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:21.417158 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:21.418632 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.419169 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.419171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.419172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.419253 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.419532 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.419534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.419535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.419720 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:21.420090 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.420421 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.420620 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":53}
I20260926 07:19:21.420763 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.410885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:36339 (local address 127.0.0.1:50164)
0926 07:19:21.411306 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.411321 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.411446 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.411728 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.411731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.412292 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:21.412414 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.412422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.413188 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.413192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.413998 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.414010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.414471 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.414475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.414568 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.415178 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:21.415194 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:21.416944 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:21.418735 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:21.418745 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:21.418752 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:21.419073 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:21.419339 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:21.419343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:21.419345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:21.419442 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:21.420102 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:21.420109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:21.420236 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.420361 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.420541 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":109,"threads_started":1}
I20260926 07:19:21.425634 30728 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:50164:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:19:21.426489 30728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:21.431792 30867 tablet_service.cc:1467] Processing CreateTablet for tablet 80301d13be674487a2d2641c64ac7c83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c4b16d76f8a241e0ba66e71ea6924901]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:19:21.431821 31007 tablet_service.cc:1467] Processing CreateTablet for tablet 80301d13be674487a2d2641c64ac7c83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c4b16d76f8a241e0ba66e71ea6924901]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:19:21.432070 31147 tablet_service.cc:1467] Processing CreateTablet for tablet 80301d13be674487a2d2641c64ac7c83 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c4b16d76f8a241e0ba66e71ea6924901]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:19:21.432232 31007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80301d13be674487a2d2641c64ac7c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:21.432245 30867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80301d13be674487a2d2641c64ac7c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:21.432438 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80301d13be674487a2d2641c64ac7c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:21.435364 31236 tablet_bootstrap.cc:492] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: Bootstrap starting.
I20260926 07:19:21.435992 31238 tablet_bootstrap.cc:492] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: Bootstrap starting.
I20260926 07:19:21.436414 31236 tablet_bootstrap.cc:654] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:21.437098 31238 tablet_bootstrap.cc:654] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:21.437129 31237 tablet_bootstrap.cc:492] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: Bootstrap starting.
I20260926 07:19:21.438105 31237 tablet_bootstrap.cc:654] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:21.439036 31238 tablet_bootstrap.cc:492] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: No bootstrap required, opened a new log
I20260926 07:19:21.439095 31236 tablet_bootstrap.cc:492] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: No bootstrap required, opened a new log
I20260926 07:19:21.439149 31238 ts_tablet_manager.cc:1397] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 07:19:21.439190 31236 ts_tablet_manager.cc:1397] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 07:19:21.439608 31238 raft_consensus.cc:348] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.439668 31236 raft_consensus.cc:348] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.439762 31238 raft_consensus.cc:374] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:21.439785 31236 raft_consensus.cc:374] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:21.439816 31236 raft_consensus.cc:729] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3959e6d4a6ee42309fc984ec57426013, State: Initialized, Role: FOLLOWER
I20260926 07:19:21.439810 31238 raft_consensus.cc:729] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338480c296d447f3b3c6c425c742149f, State: Initialized, Role: FOLLOWER
I20260926 07:19:21.439945 31236 consensus_queue.cc:260] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.439952 31238 consensus_queue.cc:260] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.440225 31236 ts_tablet_manager.cc:1428] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:19:21.440261 31244 tablet_replica.cc:439] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } } }
I20260926 07:19:21.440395 31244 tablet_replica.cc:442] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:21.441082 31238 ts_tablet_manager.cc:1428] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:19:21.440238 31243 tablet_replica.cc:439] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } } }
I20260926 07:19:21.441479 31243 tablet_replica.cc:442] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:21.442641 31237 tablet_bootstrap.cc:492] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: No bootstrap required, opened a new log
I20260926 07:19:21.442744 31237 ts_tablet_manager.cc:1397] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 07:19:21.443256 31237 raft_consensus.cc:348] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.443398 31237 raft_consensus.cc:374] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:21.443440 31237 raft_consensus.cc:729] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd81a3b82295461295e57eda01374b48, State: Initialized, Role: FOLLOWER
I20260926 07:19:21.443585 31237 consensus_queue.cc:260] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.443867 31237 ts_tablet_manager.cc:1428] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 07:19:21.444062 31242 tablet_replica.cc:439] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } } }
I20260926 07:19:21.444198 31242 tablet_replica.cc:442] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:21.501322 31244 raft_consensus.cc:482] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:21.501505 31244 raft_consensus.cc:504] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.502033 31244 leader_election.cc:288] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd81a3b82295461295e57eda01374b48 (127.28.235.193:40227), 3959e6d4a6ee42309fc984ec57426013 (127.28.235.194:45301)
I20260926 07:19:21.502421 30887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80301d13be674487a2d2641c64ac7c83" candidate_uuid: "338480c296d447f3b3c6c425c742149f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd81a3b82295461295e57eda01374b48" is_pre_election: true
I20260926 07:19:21.502535 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80301d13be674487a2d2641c64ac7c83" candidate_uuid: "338480c296d447f3b3c6c425c742149f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3959e6d4a6ee42309fc984ec57426013" is_pre_election: true
I20260926 07:19:21.502588 30887 raft_consensus.cc:2454] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 338480c296d447f3b3c6c425c742149f in term 0.
I20260926 07:19:21.502693 31027 raft_consensus.cc:2454] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 338480c296d447f3b3c6c425c742149f in term 0.
I20260926 07:19:21.502874 31100 leader_election.cc:302] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338480c296d447f3b3c6c425c742149f, fd81a3b82295461295e57eda01374b48; no voters:
I20260926 07:19:21.503093 31244 raft_consensus.cc:2781] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:21.503151 31244 raft_consensus.cc:482] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:21.503178 31244 raft_consensus.cc:3037] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:21.504189 31244 raft_consensus.cc:504] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.504524 31244 leader_election.cc:288] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [CANDIDATE]: Term 1 election: Requested vote from peers fd81a3b82295461295e57eda01374b48 (127.28.235.193:40227), 3959e6d4a6ee42309fc984ec57426013 (127.28.235.194:45301)
I20260926 07:19:21.504967 31027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80301d13be674487a2d2641c64ac7c83" candidate_uuid: "338480c296d447f3b3c6c425c742149f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3959e6d4a6ee42309fc984ec57426013"
I20260926 07:19:21.504951 30887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80301d13be674487a2d2641c64ac7c83" candidate_uuid: "338480c296d447f3b3c6c425c742149f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd81a3b82295461295e57eda01374b48"
I20260926 07:19:21.505134 31027 raft_consensus.cc:3037] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:21.505146 30887 raft_consensus.cc:3037] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:21.506069 31027 raft_consensus.cc:2454] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 338480c296d447f3b3c6c425c742149f in term 1.
I20260926 07:19:21.506100 30887 raft_consensus.cc:2454] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 338480c296d447f3b3c6c425c742149f in term 1.
I20260926 07:19:21.506376 31100 leader_election.cc:302] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 338480c296d447f3b3c6c425c742149f, 3959e6d4a6ee42309fc984ec57426013; no voters:
I20260926 07:19:21.506556 31244 raft_consensus.cc:2781] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:21.506659 31244 raft_consensus.cc:686] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [term 1 LEADER]: Becoming Leader. State: Replica: 338480c296d447f3b3c6c425c742149f, State: Running, Role: LEADER
I20260926 07:19:21.506808 31244 consensus_queue.cc:237] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } }
I20260926 07:19:21.507505 31267 tablet_replica.cc:439] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: TxnStatusTablet state changed. Reason: New leader 338480c296d447f3b3c6c425c742149f. Latest consensus state: current_term: 1 leader_uuid: "338480c296d447f3b3c6c425c742149f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } } }
I20260926 07:19:21.507639 31267 tablet_replica.cc:442] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:19:21.507934 31279 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:19:21.508103 31279 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:19:21.508639 30730 catalog_manager.cc:5495] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f reported cstate change: term changed from 0 to 1, leader changed from <none> to 338480c296d447f3b3c6c425c742149f (127.28.235.195). New cstate: current_term: 1 leader_uuid: "338480c296d447f3b3c6c425c742149f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } health_report { overall_health: HEALTHY } } }
I20260926 07:19:21.528376 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.524231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.28.235.195:39559)
0926 07:19:21.524366 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.524369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.524472 (+ 103us) server_negotiation.cc:407] Connection header received
0926 07:19:21.524672 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.524674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.524732 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.524817 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:21.525122 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.525633 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.526849 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.527061 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.527142 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.527520 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:21.527984 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:21.528077 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.528190 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.528250 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:19:21.528446 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.524132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:39559 (local address 127.0.0.1:56626)
0926 07:19:21.524395 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.524412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.524537 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.524854 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.524857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.524865 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:19:21.524996 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.525003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.525740 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.525743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.526723 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.526735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.527172 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.527176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.527271 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.527331 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:19:21.528081 (+ 750us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:19:21.528097 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.528177 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.528284 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260926 07:19:21.531790 31027 raft_consensus.cc:1261] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013 [term 1 FOLLOWER]: Refusing update from remote peer 338480c296d447f3b3c6c425c742149f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:21.531862 30887 raft_consensus.cc:1261] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48 [term 1 FOLLOWER]: Refusing update from remote peer 338480c296d447f3b3c6c425c742149f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:21.532459 31267 consensus_queue.cc:1035] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:21.532738 31244 consensus_queue.cc:1035] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:21.534343 31242 tablet_replica.cc:439] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: TxnStatusTablet state changed. Reason: New leader 338480c296d447f3b3c6c425c742149f. Latest consensus state: current_term: 1 leader_uuid: "338480c296d447f3b3c6c425c742149f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } } }
I20260926 07:19:21.534457 31242 tablet_replica.cc:442] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:21.534940 31243 tablet_replica.cc:439] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: TxnStatusTablet state changed. Reason: New leader 338480c296d447f3b3c6c425c742149f. Latest consensus state: current_term: 1 leader_uuid: "338480c296d447f3b3c6c425c742149f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } } }
I20260926 07:19:21.535058 31243 tablet_replica.cc:442] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:21.536804 31267 tablet_replica.cc:439] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "338480c296d447f3b3c6c425c742149f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } } }
I20260926 07:19:21.536934 31267 tablet_replica.cc:442] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:19:21.537725 31243 tablet_replica.cc:439] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "338480c296d447f3b3c6c425c742149f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } } }
I20260926 07:19:21.537842 31243 tablet_replica.cc:442] T 80301d13be674487a2d2641c64ac7c83 P 3959e6d4a6ee42309fc984ec57426013: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:21.541610 31244 tablet_replica.cc:439] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "338480c296d447f3b3c6c425c742149f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } } }
I20260926 07:19:21.541734 31244 tablet_replica.cc:442] T 80301d13be674487a2d2641c64ac7c83 P 338480c296d447f3b3c6c425c742149f: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:19:21.542925 31242 tablet_replica.cc:439] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "338480c296d447f3b3c6c425c742149f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 } } peers { permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 } } peers { permanent_uuid: "338480c296d447f3b3c6c425c742149f" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 39559 } } }
I20260926 07:19:21.543046 31242 tablet_replica.cc:442] T 80301d13be674487a2d2641c64ac7c83 P fd81a3b82295461295e57eda01374b48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:21.547036 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.543874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.28.235.195:39559)
0926 07:19:21.543974 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.543980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.543995 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:19:21.544034 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.544036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.544080 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.544167 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:21.544472 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.545141 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.546023 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.546264 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.546368 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.546611 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:21.546692 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:21.546766 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.546858 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.546906 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 07:19:21.557566 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.554153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:39559 (local address 127.28.235.195:49649)
0926 07:19:21.554293 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:21.554308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:21.554400 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:21.554621 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:21.554624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:21.554631 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:19:21.554743 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.554748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.555385 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.555388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.556351 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:21.556361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.556768 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.556772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:21.556870 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.556922 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:19:21.557234 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:19:21.557240 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:19:21.557312 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 07:19:21.557388 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 07:19:21.557593 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.554197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:49649 (local address 127.28.235.195:39559)
0926 07:19:21.554311 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.554314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.554351 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:19:21.554499 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.554501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.554541 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.554594 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:21.554832 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.555297 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.556457 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.556667 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.556742 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.557081 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:21.557147 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:21.557213 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.557350 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.557432 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:19:21.564955 30887 raft_consensus.cc:1261] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48 [term 1 FOLLOWER]: Refusing update from remote peer 338480c296d447f3b3c6c425c742149f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:21.564997 31027 raft_consensus.cc:1261] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013 [term 1 FOLLOWER]: Refusing update from remote peer 338480c296d447f3b3c6c425c742149f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:21.565379 31244 consensus_queue.cc:1035] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd81a3b82295461295e57eda01374b48" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 40227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:21.565523 31267 consensus_queue.cc:1035] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3959e6d4a6ee42309fc984ec57426013" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:21.609278 30728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:21.609443 30728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:21.612165 30728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6f844423d184a71b9bda7e1d6da975c: Sending DeleteTablet for 3 replicas of tablet 3548591940a9425ca44e55452ce9074f
I20260926 07:19:21.612762 30867 tablet_service.cc:1514] Processing DeleteTablet for tablet 3548591940a9425ca44e55452ce9074f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:21 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:53634
I20260926 07:19:21.612960 31260 tablet_replica.cc:330] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: stopping tablet replica
I20260926 07:19:21.612960 31007 tablet_service.cc:1514] Processing DeleteTablet for tablet 3548591940a9425ca44e55452ce9074f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:21 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:48680
I20260926 07:19:21.613111 31260 raft_consensus.cc:2229] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:21.613173 31262 tablet_replica.cc:330] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: stopping tablet replica
I20260926 07:19:21.613238 31260 raft_consensus.cc:2258] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:21.613297 31262 raft_consensus.cc:2229] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:21.613411 31262 raft_consensus.cc:2258] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:21.613411 31146 tablet_service.cc:1514] Processing DeleteTablet for tablet 3548591940a9425ca44e55452ce9074f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:21 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:56616
I20260926 07:19:21.613581 31261 tablet_replica.cc:330] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f: stopping tablet replica
I20260926 07:19:21.613695 31261 raft_consensus.cc:2229] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:21.613965 31261 raft_consensus.cc:2258] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:21.614293 31262 ts_tablet_manager.cc:1905] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:21.614635 31261 ts_tablet_manager.cc:1905] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:21.617151 31261 ts_tablet_manager.cc:1918] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:21.617161 31262 ts_tablet_manager.cc:1918] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:21.617234 31261 log.cc:1196] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/3548591940a9425ca44e55452ce9074f
I20260926 07:19:21.617233 31262 log.cc:1196] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/3548591940a9425ca44e55452ce9074f
I20260926 07:19:21.617501 31262 ts_tablet_manager.cc:1939] T 3548591940a9425ca44e55452ce9074f P 3959e6d4a6ee42309fc984ec57426013: Deleting consensus metadata
I20260926 07:19:21.617501 31261 ts_tablet_manager.cc:1939] T 3548591940a9425ca44e55452ce9074f P 338480c296d447f3b3c6c425c742149f: Deleting consensus metadata
I20260926 07:19:21.618220 30717 catalog_manager.cc:4841] TS 338480c296d447f3b3c6c425c742149f (127.28.235.195:39559): tablet 3548591940a9425ca44e55452ce9074f (table test-table [id=2a78d324358146a796aa44e2a748013f]) successfully deleted
I20260926 07:19:21.618490 30716 catalog_manager.cc:4841] TS 3959e6d4a6ee42309fc984ec57426013 (127.28.235.194:45301): tablet 3548591940a9425ca44e55452ce9074f (table test-table [id=2a78d324358146a796aa44e2a748013f]) successfully deleted
I20260926 07:19:21.619984 31260 ts_tablet_manager.cc:1905] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:21.622985 31260 ts_tablet_manager.cc:1918] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:21.623220 31260 log.cc:1196] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/3548591940a9425ca44e55452ce9074f
I20260926 07:19:21.623579 31260 ts_tablet_manager.cc:1939] T 3548591940a9425ca44e55452ce9074f P fd81a3b82295461295e57eda01374b48: Deleting consensus metadata
I20260926 07:19:21.624344 30716 catalog_manager.cc:4841] TS fd81a3b82295461295e57eda01374b48 (127.28.235.193:40227): tablet 3548591940a9425ca44e55452ce9074f (table test-table [id=2a78d324358146a796aa44e2a748013f]) successfully deleted
Sep 26 07:19:21 dist-test-slave-7zlw krb5kdc[30685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:19:21.640205 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.621126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.28.235.193:40227)
0926 07:19:21.621266 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.621271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.629396 (+ 8125us) server_negotiation.cc:407] Connection header received
0926 07:19:21.629509 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.629512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.629570 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.629678 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:21.630545 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.631408 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.632296 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.632676 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.632818 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.636397 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:21.636407 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:21.636419 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:21.636444 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:21.638321 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.638849 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.638853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.638855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.638933 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.639222 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.639224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.639225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.639475 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:21.639619 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.639916 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.640024 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 07:19:21.640657 30927 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53652
I20260926 07:19:21.649863 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:21.642544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.28.235.254:36339)
0926 07:19:21.642699 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:21.642703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:21.642722 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:19:21.642843 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:21.642847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:21.642902 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:21.642984 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:21.643612 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.644137 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.644918 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:21.645240 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:21.645340 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:21.646715 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:21.646719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:21.646722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:21.646748 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:21.648128 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.648563 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.648566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.648568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.648656 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:21.648924 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:21.648928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:21.648930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:21.649174 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:21.649315 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:21.649556 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 07:19:21.649681 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 07:19:21.650324 30728 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50178
I20260926 07:19:21.651775 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 30802
I20260926 07:19:21.663465 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 30942
I20260926 07:19:21.670101 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 31082
I20260926 07:19:21.676998 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 30698
2026-09-26T07:19:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2769 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:21 dist-test-slave-7zlw krb5kdc[31297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:21 dist-test-slave-7zlw krb5kdc[31297](info): set up 2 sockets
Sep 26 07:19:21 dist-test-slave-7zlw krb5kdc[31297](info): commencing operation
krb5kdc: starting...
W20260926 07:19:22.740971 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:22 dist-test-slave-7zlw krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:22.870366 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34877
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:22.982995 31310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:22.983345 31310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:22.983423 31310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:22.987557 31310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:22.987654 31310 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:19:22.987689 31310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:22.987751 31310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:22.987804 31310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:22.987846 31310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:22.993330 31310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:34877
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:22.994453 31310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:22.995543 31310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:23.005538 31316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:23.006162 31310 server_base.cc:1034] running on GCE node
W20260926 07:19:23.007789 31318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:23.008687 31315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:23.009511 31310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:23.011031 31310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:23.012286 31310 hybrid_clock.cc:648] HybridClock initialized: now 1790407163012244 us; error 45 us; skew 500 ppm
Sep 26 07:19:23 dist-test-slave-7zlw krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:23.015337 31310 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:23.016283 31310 webserver.cc:460] Webserver started at http://127.28.235.254:38221/ using document root <none> and password file <none>
I20260926 07:19:23.016824 31310 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:23.016880 31310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:23.017112 31310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:23.019176 31310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "53bfeee7da2541988ef3376d3bd0417c"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "841cd7667058ffddeb2a2eab1ea29c02"
server_key_iv: "78c42659cd93bc78085aabe4be4f20c2"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.019690 31310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "53bfeee7da2541988ef3376d3bd0417c"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "841cd7667058ffddeb2a2eab1ea29c02"
server_key_iv: "78c42659cd93bc78085aabe4be4f20c2"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.023559 31310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:19:23.026201 31325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:23.027606 31310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:23.027714 31310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "53bfeee7da2541988ef3376d3bd0417c"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "841cd7667058ffddeb2a2eab1ea29c02"
server_key_iv: "78c42659cd93bc78085aabe4be4f20c2"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.027812 31310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:23.067284 31310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:23.070922 31310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:23.071107 31310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:23.079651 31310 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:34877
I20260926 07:19:23.079670 31387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:34877 every 8 connection(s)
I20260926 07:19:23.080638 31310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:23.083676 31388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:23.087438 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 31310
I20260926 07:19:23.087589 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:23.087893 29615 external_mini_cluster.cc:1428] Setting key ae36fd4c5a72d5f7c10004813488b628
I20260926 07:19:23.089658 31388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c: Bootstrap starting.
I20260926 07:19:23.092438 31388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:23.093272 31388 log.cc:824] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:23.095078 31388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c: No bootstrap required, opened a new log
Sep 26 07:19:23 dist-test-slave-7zlw krb5kdc[31297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:23.098762 31388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bfeee7da2541988ef3376d3bd0417c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34877 } }
I20260926 07:19:23.099020 31388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:23.099078 31388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53bfeee7da2541988ef3376d3bd0417c, State: Initialized, Role: FOLLOWER
I20260926 07:19:23.099501 31388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bfeee7da2541988ef3376d3bd0417c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34877 } }
I20260926 07:19:23.099649 31388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:23.099728 31388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:23.099869 31388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:23.100813 31388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bfeee7da2541988ef3376d3bd0417c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34877 } }
I20260926 07:19:23.101238 31388 leader_election.cc:302] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53bfeee7da2541988ef3376d3bd0417c; no voters:
I20260926 07:19:23.101608 31388 leader_election.cc:288] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:23.101727 31393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:23.101977 31393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [term 1 LEADER]: Becoming Leader. State: Replica: 53bfeee7da2541988ef3376d3bd0417c, State: Running, Role: LEADER
I20260926 07:19:23.102312 31393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bfeee7da2541988ef3376d3bd0417c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34877 } }
I20260926 07:19:23.102973 31388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:23.103137 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.089455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.28.235.254:34877)
0926 07:19:23.089922 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.089945 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.089986 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:19:23.090793 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.090819 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.091279 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.091639 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:23.092474 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.093605 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.094433 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.094901 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.095068 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.097498 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:23.097503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:23.097516 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:23.097543 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:23.099516 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.100036 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.100043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.100053 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.100249 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.100553 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.100556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.100558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.101067 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:23.101292 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.101601 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.102134 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":121,"threads_started":1}
I20260926 07:19:23.103940 31395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53bfeee7da2541988ef3376d3bd0417c. Latest consensus state: current_term: 1 leader_uuid: "53bfeee7da2541988ef3376d3bd0417c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bfeee7da2541988ef3376d3bd0417c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34877 } } }
I20260926 07:19:23.104072 31395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:23.104377 31394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53bfeee7da2541988ef3376d3bd0417c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bfeee7da2541988ef3376d3bd0417c" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34877 } } }
I20260926 07:19:23.104476 31394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:23.105540 31402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:23.108198 31402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:23.113804 31402 catalog_manager.cc:1347] Generated new cluster ID: 5b5515b138ff4605aa218ada0ad18ca4
I20260926 07:19:23.113886 31402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:23.134941 31402 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:23.136273 31402 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:23.146061 31402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c: Generated new TSK 0
I20260926 07:19:23.146963 31402 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:19:23.184487 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34877
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:23.300273 31414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:23.301390 31414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:23.301755 31414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:23.306524 31414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:23.306602 31414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:23.306648 31414 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:19:23.306700 31414 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:19:23.306762 31414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:23.312649 31414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.235.254:34877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:23.313933 31414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:23.315054 31414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:23.325464 31420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:23.325475 31419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:23.325479 31422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:23.325693 31414 server_base.cc:1034] running on GCE node
I20260926 07:19:23.326169 31414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:23.326886 31414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:23.328101 31414 hybrid_clock.cc:648] HybridClock initialized: now 1790407163328082 us; error 28 us; skew 500 ppm
Sep 26 07:19:23 dist-test-slave-7zlw krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:23.331290 31414 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:19:23.332324 31414 webserver.cc:460] Webserver started at http://127.28.235.193:40723/ using document root <none> and password file <none>
I20260926 07:19:23.332908 31414 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:23.332976 31414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:23.333244 31414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:23.335439 31414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "cd5c7c6e1d09c2bcd0047997f0295d08"
server_key_iv: "39faeb6ab06cb6999e03ab04ac3b4f8f"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.336021 31414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "cd5c7c6e1d09c2bcd0047997f0295d08"
server_key_iv: "39faeb6ab06cb6999e03ab04ac3b4f8f"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.340152 31414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:19:23.342890 31429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:23.344193 31414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:19:23.344350 31414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "cd5c7c6e1d09c2bcd0047997f0295d08"
server_key_iv: "39faeb6ab06cb6999e03ab04ac3b4f8f"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.344472 31414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:23.357175 31414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:23.360368 31414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:23.360574 31414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:23.362238 31414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:23.362291 31414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:23.362355 31414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:23.362387 31414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:19:23 dist-test-slave-7zlw krb5kdc[31297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:23.374061 31414 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:38217
I20260926 07:19:23.374101 31544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:38217 every 8 connection(s)
I20260926 07:19:23.375152 31414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:23.376297 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.363115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:46645 (local address 127.28.235.254:34877)
0926 07:19:23.363298 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.363304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.364143 (+ 839us) server_negotiation.cc:407] Connection header received
0926 07:19:23.365300 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.365307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.365380 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.365443 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:23.366875 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.367506 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.368666 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.368965 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.369123 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.372359 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:23.372365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:23.372372 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:23.372409 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:23.374183 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.374698 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.374701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.374703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.374789 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.375112 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.375118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.375123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.375355 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:23.375453 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.376043 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.376178 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:19:23.377314 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.363509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34877 (local address 127.28.235.193:46645)
0926 07:19:23.363971 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.364022 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.364997 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.365828 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.365835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.366172 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:23.366720 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.366733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.367636 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.367639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.368534 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.368543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.369180 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.369184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.369356 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.370301 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:23.370322 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:23.372093 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:23.374312 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.374318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.374337 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.374593 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.374884 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.374886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.374888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.375018 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.375469 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:23.375474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:23.375781 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.375988 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.376359 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":134,"threads_started":1}
I20260926 07:19:23.378798 31545 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34877
I20260926 07:19:23.378962 31545 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:23.379539 31545 heartbeater.cc:511] Master 127.28.235.254:34877 requested a full tablet report, sending...
I20260926 07:19:23.381254 31342 ts_manager.cc:194] Registered new tserver with Master: 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193:38217)
I20260926 07:19:23.381309 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 31414
I20260926 07:19:23.381413 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:23.381750 29615 external_mini_cluster.cc:1428] Setting key e77656443723e896fa2e53bdda037722
I20260926 07:19:23.382781 31342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:46645
I20260926 07:19:23.389178 31551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.380787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34877 (local address 127.28.235.193:59781)
0926 07:19:23.381118 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.381134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.381231 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.381587 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.381591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.381807 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:23.381933 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.381942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.383024 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.383027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.383829 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.383837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.384291 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.384295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.384449 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.384949 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:23.384960 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:23.385495 (+ 535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:23.387575 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.387578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.387579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.387783 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.388132 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.388136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.388142 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.388230 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.388609 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:23.388612 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:23.388705 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.388809 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.388950 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":20,"thread_start_us":86,"threads_started":1}
I20260926 07:19:23.389169 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.380901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:59781 (local address 127.28.235.254:34877)
0926 07:19:23.381053 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.381057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.381214 (+ 157us) server_negotiation.cc:407] Connection header received
0926 07:19:23.381331 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.381335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.381389 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.381467 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:23.382055 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.382855 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.383941 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.384181 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.384286 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.385667 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:23.385669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:23.385672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:23.385694 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:23.387454 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.387899 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.387904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.387906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.387987 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.388326 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.388329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.388331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.388514 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:23.388620 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.388862 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.388980 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:19:23.412882 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34877
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:23.532928 31554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:23.533309 31554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:23.533380 31554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:23.537695 31554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:23.537765 31554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:23.537809 31554 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:19:23.537860 31554 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:19:23.537927 31554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:23.543509 31554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.235.254:34877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:23.544596 31554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:23.545678 31554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:23.555563 31559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:23.555595 31562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:23.555675 31554 server_base.cc:1034] running on GCE node
W20260926 07:19:23.555563 31560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:23.556293 31554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:23.556978 31554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:23.558220 31554 hybrid_clock.cc:648] HybridClock initialized: now 1790407163558213 us; error 43 us; skew 500 ppm
Sep 26 07:19:23 dist-test-slave-7zlw krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:23.561280 31554 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:19:23.562208 31554 webserver.cc:460] Webserver started at http://127.28.235.194:45879/ using document root <none> and password file <none>
I20260926 07:19:23.562879 31554 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:23.562947 31554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:23.563189 31554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:23.565541 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "f0e0765e2f434df0b3e79e53f2c0e62a"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "dd51b6779838126b6a57cedc3bcbeee4"
server_key_iv: "62947b74c713b570ac8ad4aa0342b2af"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.566147 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "f0e0765e2f434df0b3e79e53f2c0e62a"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "dd51b6779838126b6a57cedc3bcbeee4"
server_key_iv: "62947b74c713b570ac8ad4aa0342b2af"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.570448 31554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:23.572930 31569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:23.574502 31554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:23.574636 31554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "f0e0765e2f434df0b3e79e53f2c0e62a"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "dd51b6779838126b6a57cedc3bcbeee4"
server_key_iv: "62947b74c713b570ac8ad4aa0342b2af"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.574754 31554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:23.585589 31554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:23.588863 31554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:23.589102 31554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:23.590567 31554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:23.590641 31554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:23.590736 31554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:23.590781 31554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:19:23 dist-test-slave-7zlw krb5kdc[31297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:23.602455 31554 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:37263
I20260926 07:19:23.602568 31684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:37263 every 8 connection(s)
I20260926 07:19:23.603790 31554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:23.605804 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.591710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:43317 (local address 127.28.235.254:34877)
0926 07:19:23.591837 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.591841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.592955 (+ 1114us) server_negotiation.cc:407] Connection header received
0926 07:19:23.594235 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.594240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.594308 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.594417 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:23.595678 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.596593 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.598020 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.598442 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.598572 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.601448 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:23.601455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:23.601461 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:23.601499 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:23.603368 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.603949 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.603953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.603954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.604036 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.604480 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.604486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.604489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.604789 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:23.604897 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.605501 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.605634 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:19:23.606570 31578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.592256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34877 (local address 127.28.235.194:43317)
0926 07:19:23.592789 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.592833 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.593997 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.594628 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.594636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.594995 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:23.595539 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.595553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.596729 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.596732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.597913 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.597922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.598607 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.598611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.598843 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.599597 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:23.599616 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:23.601277 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:23.603499 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.603505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.603537 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.603829 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.604160 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.604165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.604167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.604381 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.604896 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:23.604902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:23.605256 (+ 354us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.605441 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.605773 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":140,"threads_started":1}
I20260926 07:19:23.607779 31685 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34877
I20260926 07:19:23.607932 31685 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:23.608436 31685 heartbeater.cc:511] Master 127.28.235.254:34877 requested a full tablet report, sending...
I20260926 07:19:23.609447 31341 ts_manager.cc:194] Registered new tserver with Master: f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263)
I20260926 07:19:23.609941 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 31554
I20260926 07:19:23.610037 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:23.610256 31341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:43317
I20260926 07:19:23.610316 29615 external_mini_cluster.cc:1428] Setting key f77b9c5db2123841407de4f611e1c4ce
I20260926 07:19:23.617976 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.610039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:54089 (local address 127.28.235.254:34877)
0926 07:19:23.610548 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.610552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.610565 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:19:23.610599 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.610601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.610645 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.610725 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:23.611338 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.612055 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.612925 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.613284 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.613367 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.614714 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:23.614717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:23.614719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:23.614740 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:23.616148 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.616653 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.616655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.616656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.616725 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.617058 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.617064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.617066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.617309 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:23.617387 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.617738 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.617861 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443}
I20260926 07:19:23.618044 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.609997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34877 (local address 127.28.235.194:54089)
0926 07:19:23.610250 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.610263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.610333 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.610863 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.610866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.611072 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:23.611185 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.611192 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.612197 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.612209 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.612808 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.612817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.613397 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.613402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.613521 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.614113 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:23.614131 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:23.614574 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:23.616258 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.616263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.616269 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.616556 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.616812 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.616819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.616823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.616928 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.617408 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:23.617413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:23.617514 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.617635 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.617847 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:19:23.641501 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34877
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:23.753741 31694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:23.754011 31694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:23.754060 31694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:23.758277 31694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:23.758360 31694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:23.758428 31694 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:19:23.758463 31694 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:19:23.758535 31694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:23.763990 31694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.235.254:34877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:23.765136 31694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:23.766176 31694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:23.776124 31702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:23.776153 31699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:23.776139 31700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:23.776589 31694 server_base.cc:1034] running on GCE node
I20260926 07:19:23.777098 31694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:23.777765 31694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:23.778946 31694 hybrid_clock.cc:648] HybridClock initialized: now 1790407163778901 us; error 92 us; skew 500 ppm
Sep 26 07:19:23 dist-test-slave-7zlw krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:23.781870 31694 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:19:23.782757 31694 webserver.cc:460] Webserver started at http://127.28.235.195:36759/ using document root <none> and password file <none>
I20260926 07:19:23.783301 31694 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:23.783358 31694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:23.783521 31694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:23.785548 31694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "e1f80d1526ce45518d8a7886a9dcc49e"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "932ff749280b73e1de0b37849506b939"
server_key_iv: "585a542aedcf458585b53320b9f908fb"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.786041 31694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "e1f80d1526ce45518d8a7886a9dcc49e"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "932ff749280b73e1de0b37849506b939"
server_key_iv: "585a542aedcf458585b53320b9f908fb"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.789808 31694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:19:23.792273 31709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:23.793777 31694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:23.793933 31694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "e1f80d1526ce45518d8a7886a9dcc49e"
format_stamp: "Formatted at 2026-09-26 07:19:23 on dist-test-slave-7zlw"
server_key: "932ff749280b73e1de0b37849506b939"
server_key_iv: "585a542aedcf458585b53320b9f908fb"
server_key_version: "encryptionkey@0"
I20260926 07:19:23.794066 31694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:23.812727 31694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:23.815970 31694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:23.816175 31694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:23.817786 31694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:23.817852 31694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:23.817917 31694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:23.817965 31694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:19:23 dist-test-slave-7zlw krb5kdc[31297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:23.829398 31694 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:34337
I20260926 07:19:23.829681 31824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:34337 every 8 connection(s)
I20260926 07:19:23.830580 31694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:23.832545 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.818723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:50117 (local address 127.28.235.254:34877)
0926 07:19:23.818853 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.818857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.820055 (+ 1198us) server_negotiation.cc:407] Connection header received
0926 07:19:23.821169 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.821173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.821240 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.821350 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:23.822702 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.823253 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.824626 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.824888 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.825050 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.828325 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:23.828333 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:23.828336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:23.828371 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:23.830233 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.830776 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.830780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.830782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.830890 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.831347 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.831350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.831352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.831582 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:23.831698 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.832249 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.832381 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:19:23.833421 31718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.819271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34877 (local address 127.28.235.195:50117)
0926 07:19:23.819829 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.819874 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.820874 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.821539 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.821547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.821909 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:23.822535 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.822552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.823389 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.823396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.824486 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.824498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.825100 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.825105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.825290 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.826041 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:23.826067 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:23.828136 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:23.830353 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.830357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.830369 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.830667 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.830981 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.830985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.830987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.831251 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.831694 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:23.831701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:23.832004 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.832211 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.832544 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":136,"threads_started":1}
I20260926 07:19:23.834700 31825 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34877
I20260926 07:19:23.834858 31825 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:23.835357 31825 heartbeater.cc:511] Master 127.28.235.254:34877 requested a full tablet report, sending...
I20260926 07:19:23.836383 31342 ts_manager.cc:194] Registered new tserver with Master: e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:23.837129 31342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:50117
I20260926 07:19:23.838105 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 31694
I20260926 07:19:23.838239 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:23.838551 29615 external_mini_cluster.cc:1428] Setting key b905dd63022159cbf4211daebf2c9313
I20260926 07:19:23.840583 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:19:23.845148 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.836870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:41743 (local address 127.28.235.254:34877)
0926 07:19:23.836996 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.837002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.837166 (+ 164us) server_negotiation.cc:407] Connection header received
0926 07:19:23.837329 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.837331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.837372 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.837458 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:23.838088 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.838825 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.839948 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.840275 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.840383 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.841643 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:23.841646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:23.841648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:23.841671 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:23.843493 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.843901 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.843904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.843906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.844004 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.844252 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.844254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.844255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.844457 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:23.844536 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.844861 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.844956 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:19:23.845410 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.836812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34877 (local address 127.28.235.195:41743)
0926 07:19:23.837096 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.837112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.837224 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.837562 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.837568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.837804 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:23.837946 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.837954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.838991 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.838994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.839839 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.839854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.840390 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.840395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.840489 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.841041 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:23.841058 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:23.841508 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:23.843603 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.843605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.843607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.843806 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.844096 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:23.844098 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:23.844099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:23.844169 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:23.844570 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:23.844576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:23.844692 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.844811 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.844985 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
Sep 26 07:19:23 dist-test-slave-7zlw krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:19:23 dist-test-slave-7zlw krb5kdc[31297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:23.868808 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.860433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.28.235.254:34877)
0926 07:19:23.860570 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.860574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.860585 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:19:23.860626 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.860629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.860668 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.860760 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:23.861536 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.862068 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.862948 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.863167 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.863286 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.865853 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:23.865856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:23.865858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:23.865895 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:23.867289 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.867690 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.867693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.867694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.867767 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:23.868077 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:23.868079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:23.868080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:23.868266 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:23.868362 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.868586 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.868680 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:19:23.871999 31342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:19:23.875203 31342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:23.891099 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.884835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:34337 (local address 127.0.0.1:38974)
0926 07:19:23.885914 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.885927 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.886026 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.886622 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.886628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.886657 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:23.886759 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.886766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.888057 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.888061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.889557 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.889568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.890725 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.890729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.890821 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.890840 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.890907 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.890958 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":80,"thread_start_us":89,"threads_started":1}
I20260926 07:19:23.891196 31847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.885331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.28.235.195:34337)
0926 07:19:23.885766 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.885780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.886012 (+ 232us) server_negotiation.cc:407] Connection header received
0926 07:19:23.886137 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.886143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.886471 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.886622 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:23.886876 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.887957 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.889706 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.890517 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.890670 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.890804 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.890936 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.891037 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":107,"threads_started":1}
I20260926 07:19:23.891414 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.885318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.28.235.193:38217)
0926 07:19:23.886289 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.886295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.886316 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:19:23.886386 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.886389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.886529 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.886682 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:23.887017 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.887759 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.889738 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.890369 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.890503 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.890650 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.891166 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.891291 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":91,"threads_started":1}
I20260926 07:19:23.891099 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.884605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38217 (local address 127.0.0.1:48658)
0926 07:19:23.885776 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.885790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.885875 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.886811 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.886814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.886834 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:23.886906 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.886916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.887873 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.887876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.889585 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.889597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.890500 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.890504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.890621 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.890658 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.890782 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.890921 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"mutex_wait_us":183,"thread_start_us":116,"threads_started":1}
I20260926 07:19:23.891381 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.884517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:37263 (local address 127.0.0.1:37700)
0926 07:19:23.885147 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.885199 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.885349 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.885982 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.885989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.886012 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:23.886168 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.886174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.887794 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.887800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.889695 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.889702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.890885 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.890890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.890987 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.891009 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.891077 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.891157 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":220,"thread_start_us":63,"threads_started":1}
I20260926 07:19:23.891736 31844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.884640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.28.235.194:37263)
0926 07:19:23.885463 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.885478 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.885502 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:19:23.885575 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.885580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.885807 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.886346 (+ 539us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:23.886490 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.887670 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.889835 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.890607 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.890726 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.890823 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.891483 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.891593 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":109,"threads_started":1}
I20260926 07:19:23.894660 31479 tablet_service.cc:1467] Processing CreateTablet for tablet da33f64a04d641b29e143a512b957992 (DEFAULT_TABLE table=test-table [id=3c4f443db0414e709c37127022181aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:23.894685 31619 tablet_service.cc:1467] Processing CreateTablet for tablet da33f64a04d641b29e143a512b957992 (DEFAULT_TABLE table=test-table [id=3c4f443db0414e709c37127022181aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:23.894685 31759 tablet_service.cc:1467] Processing CreateTablet for tablet da33f64a04d641b29e143a512b957992 (DEFAULT_TABLE table=test-table [id=3c4f443db0414e709c37127022181aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:23.895736 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da33f64a04d641b29e143a512b957992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:23.895740 31619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da33f64a04d641b29e143a512b957992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:23.895749 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da33f64a04d641b29e143a512b957992. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:23.902423 31848 tablet_bootstrap.cc:492] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Bootstrap starting.
I20260926 07:19:23.904299 31849 tablet_bootstrap.cc:492] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: Bootstrap starting.
I20260926 07:19:23.904631 31850 tablet_bootstrap.cc:492] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: Bootstrap starting.
I20260926 07:19:23.905221 31848 tablet_bootstrap.cc:654] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:23.906159 31848 log.cc:824] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:23.906672 31849 tablet_bootstrap.cc:654] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:23.906682 31850 tablet_bootstrap.cc:654] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:23.907588 31849 log.cc:824] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:23.907588 31850 log.cc:824] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:23.908313 31848 tablet_bootstrap.cc:492] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: No bootstrap required, opened a new log
I20260926 07:19:23.908563 31848 ts_tablet_manager.cc:1397] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260926 07:19:23.910095 31850 tablet_bootstrap.cc:492] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: No bootstrap required, opened a new log
I20260926 07:19:23.910094 31849 tablet_bootstrap.cc:492] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: No bootstrap required, opened a new log
I20260926 07:19:23.910288 31849 ts_tablet_manager.cc:1397] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:19:23.910288 31850 ts_tablet_manager.cc:1397] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:19:23.912045 31848 raft_consensus.cc:348] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:23.912315 31848 raft_consensus.cc:374] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:23.912386 31848 raft_consensus.cc:729] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2f3a02f94a4dbab3e5a06b392f7efd, State: Initialized, Role: FOLLOWER
I20260926 07:19:23.912866 31848 consensus_queue.cc:260] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:23.913544 31545 heartbeater.cc:503] Master 127.28.235.254:34877 was elected leader, sending a full tablet report...
I20260926 07:19:23.914320 31849 raft_consensus.cc:348] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:23.914320 31850 raft_consensus.cc:348] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:23.914610 31850 raft_consensus.cc:374] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:23.914678 31849 raft_consensus.cc:374] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:23.914724 31850 raft_consensus.cc:729] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f80d1526ce45518d8a7886a9dcc49e, State: Initialized, Role: FOLLOWER
I20260926 07:19:23.914779 31849 raft_consensus.cc:729] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e0765e2f434df0b3e79e53f2c0e62a, State: Initialized, Role: FOLLOWER
I20260926 07:19:23.915241 31849 consensus_queue.cc:260] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:23.915228 31850 consensus_queue.cc:260] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:23.915519 31848 ts_tablet_manager.cc:1428] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260926 07:19:23.916239 31850 ts_tablet_manager.cc:1428] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:19:23.916239 31849 ts_tablet_manager.cc:1428] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:19:23.916590 31825 heartbeater.cc:503] Master 127.28.235.254:34877 was elected leader, sending a full tablet report...
I20260926 07:19:23.916592 31685 heartbeater.cc:503] Master 127.28.235.254:34877 was elected leader, sending a full tablet report...
I20260926 07:19:23.960644 31854 raft_consensus.cc:482] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:23.960877 31854 raft_consensus.cc:504] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:23.962312 31854 leader_election.cc:288] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263), e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:23.967869 31847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.962716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:38577 (local address 127.28.235.195:34337)
0926 07:19:23.962868 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.962872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.963131 (+ 259us) server_negotiation.cc:407] Connection header received
0926 07:19:23.963365 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.963368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.963423 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.963473 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:23.963808 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.964678 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.966423 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.967060 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.967198 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.967251 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.967642 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.967719 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:19:23.967869 31844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.962643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:49811 (local address 127.28.235.194:37263)
0926 07:19:23.962802 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:23.962808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:23.962988 (+ 180us) server_negotiation.cc:407] Connection header received
0926 07:19:23.963173 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:23.963177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:23.963240 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:23.963315 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:23.963641 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.964542 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.966423 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.967073 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.967198 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.967251 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:23.967642 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 07:19:23.967719 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:19:23.968016 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.962462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:37263 (local address 127.28.235.193:49811)
0926 07:19:23.962888 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.962901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.963045 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.963348 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.963351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.963369 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:23.963517 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.963522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.964685 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.964687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.966278 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.966290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.967200 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.967204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.967417 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.967532 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.967613 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.967803 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":250,"thread_start_us":137,"threads_started":1}
I20260926 07:19:23.968016 31858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:23.962541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:34337 (local address 127.28.235.193:38577)
0926 07:19:23.963042 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:23.963056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:23.963219 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:23.963597 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:23.963599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:23.963612 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:23.963702 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.963706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.964783 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.964786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.966296 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:23.966302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:23.967204 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:23.967206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:23.967414 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:23.967532 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 07:19:23.967613 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:19:23.967803 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":320,"thread_start_us":135,"threads_started":1}
I20260926 07:19:23.968772 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da33f64a04d641b29e143a512b957992" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" is_pre_election: true
I20260926 07:19:23.968775 31639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da33f64a04d641b29e143a512b957992" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" is_pre_election: true
I20260926 07:19:23.969120 31639 raft_consensus.cc:2454] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 0.
I20260926 07:19:23.969118 31778 raft_consensus.cc:2454] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 0.
I20260926 07:19:23.969641 31430 leader_election.cc:302] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f3a02f94a4dbab3e5a06b392f7efd, e1f80d1526ce45518d8a7886a9dcc49e; no voters:
I20260926 07:19:23.969964 31854 raft_consensus.cc:2781] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:23.970050 31854 raft_consensus.cc:482] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:23.970100 31854 raft_consensus.cc:3037] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:23.971185 31854 raft_consensus.cc:504] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:23.971585 31854 leader_election.cc:288] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 election: Requested vote from peers f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263), e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:23.972013 31639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da33f64a04d641b29e143a512b957992" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a"
I20260926 07:19:23.972159 31639 raft_consensus.cc:3037] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:23.973230 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da33f64a04d641b29e143a512b957992" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f80d1526ce45518d8a7886a9dcc49e"
I20260926 07:19:23.973376 31778 raft_consensus.cc:3037] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:23.973747 31639 raft_consensus.cc:2454] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 1.
I20260926 07:19:23.974102 31433 leader_election.cc:302] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f3a02f94a4dbab3e5a06b392f7efd, f0e0765e2f434df0b3e79e53f2c0e62a; no voters:
I20260926 07:19:23.974288 31854 raft_consensus.cc:2781] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:23.974565 31854 raft_consensus.cc:686] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 LEADER]: Becoming Leader. State: Replica: 1e2f3a02f94a4dbab3e5a06b392f7efd, State: Running, Role: LEADER
I20260926 07:19:23.974675 31778 raft_consensus.cc:2454] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 1.
I20260926 07:19:23.974965 31854 consensus_queue.cc:237] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:23.977944 31340 catalog_manager.cc:5495] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193). New cstate: current_term: 1 leader_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:19:24 dist-test-slave-7zlw krb5kdc[31297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407163, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:24.015511 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.006856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.28.235.254:34877)
0926 07:19:24.007103 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:24.007108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:24.007235 (+ 127us) server_negotiation.cc:407] Connection header received
0926 07:19:24.007420 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:24.007423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:24.007459 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:24.007576 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:24.008289 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.008869 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.009669 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.009885 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.009993 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.012224 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:24.012228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:24.012230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:24.012261 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:24.013720 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:24.014240 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:24.014243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:24.014245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:24.014320 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:24.014608 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:24.014612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:24.014614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:24.014801 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:24.014903 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:24.015176 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:19:24.015377 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":65}
I20260926 07:19:24.015520 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.006801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34877 (local address 127.0.0.1:40014)
0926 07:19:24.007138 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:24.007159 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:24.007307 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:24.007613 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:24.007616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:24.008076 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:24.008168 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.008175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.008968 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.008974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.009556 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.009564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.009996 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.009999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.010071 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.010653 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:24.010665 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:24.012088 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:24.013855 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:24.013868 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:24.013878 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:24.014130 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:24.014403 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:24.014408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:24.014410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:24.014488 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:24.014903 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:24.014912 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:24.015016 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 07:19:24.015131 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 07:19:24.015313 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
I20260926 07:19:24.019755 31340 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:40014:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:19:24.020443 31340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:24.026132 31479 tablet_service.cc:1467] Processing CreateTablet for tablet 593e4ea5bcf34d3faf75ede7bb736674 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e70f8e505184e8994dfb0d3c2bee7db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:19:24.026121 31619 tablet_service.cc:1467] Processing CreateTablet for tablet 593e4ea5bcf34d3faf75ede7bb736674 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e70f8e505184e8994dfb0d3c2bee7db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:19:24.026476 31759 tablet_service.cc:1467] Processing CreateTablet for tablet 593e4ea5bcf34d3faf75ede7bb736674 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4e70f8e505184e8994dfb0d3c2bee7db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:19:24.026948 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 593e4ea5bcf34d3faf75ede7bb736674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:24.026949 31619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 593e4ea5bcf34d3faf75ede7bb736674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:24.027034 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 593e4ea5bcf34d3faf75ede7bb736674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:24.029872 31850 tablet_bootstrap.cc:492] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: Bootstrap starting.
I20260926 07:19:24.029878 31849 tablet_bootstrap.cc:492] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: Bootstrap starting.
I20260926 07:19:24.029878 31848 tablet_bootstrap.cc:492] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Bootstrap starting.
I20260926 07:19:24.030871 31850 tablet_bootstrap.cc:654] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:24.030911 31849 tablet_bootstrap.cc:654] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:24.031203 31848 tablet_bootstrap.cc:654] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:24.032398 31850 tablet_bootstrap.cc:492] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: No bootstrap required, opened a new log
I20260926 07:19:24.032395 31849 tablet_bootstrap.cc:492] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: No bootstrap required, opened a new log
I20260926 07:19:24.032395 31848 tablet_bootstrap.cc:492] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: No bootstrap required, opened a new log
I20260926 07:19:24.032477 31849 ts_tablet_manager.cc:1397] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:24.032477 31848 ts_tablet_manager.cc:1397] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 07:19:24.032505 31850 ts_tablet_manager.cc:1397] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:24.032898 31850 raft_consensus.cc:348] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.032985 31850 raft_consensus.cc:374] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:24.032938 31848 raft_consensus.cc:348] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.033006 31850 raft_consensus.cc:729] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f80d1526ce45518d8a7886a9dcc49e, State: Initialized, Role: FOLLOWER
I20260926 07:19:24.033074 31848 raft_consensus.cc:374] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:24.032938 31849 raft_consensus.cc:348] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.033113 31848 raft_consensus.cc:729] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2f3a02f94a4dbab3e5a06b392f7efd, State: Initialized, Role: FOLLOWER
I20260926 07:19:24.033123 31849 raft_consensus.cc:374] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:24.033167 31849 raft_consensus.cc:729] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e0765e2f434df0b3e79e53f2c0e62a, State: Initialized, Role: FOLLOWER
I20260926 07:19:24.033191 31850 consensus_queue.cc:260] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.033257 31848 consensus_queue.cc:260] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.033281 31849 consensus_queue.cc:260] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.033470 31850 ts_tablet_manager.cc:1428] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:19:24.033514 31849 ts_tablet_manager.cc:1428] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:24.033514 31848 ts_tablet_manager.cc:1428] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:24.033443 31855 tablet_replica.cc:439] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } } }
I20260926 07:19:24.033586 31855 tablet_replica.cc:442] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:24.033759 31854 tablet_replica.cc:439] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } } }
I20260926 07:19:24.033900 31854 tablet_replica.cc:442] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:24.034091 31856 tablet_replica.cc:439] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } } }
I20260926 07:19:24.034226 31856 tablet_replica.cc:442] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 07:19:24.081693 31826 tablet.cc:2369] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:19:24.105347 31686 tablet.cc:2369] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:19:24.126607 31546 tablet.cc:2369] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:19:24.170349 31856 raft_consensus.cc:482] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:24.170504 31856 raft_consensus.cc:504] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.171627 31856 leader_election.cc:288] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193:38217), e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:24.175791 31847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.172172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:35283 (local address 127.28.235.195:34337)
0926 07:19:24.172323 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:24.172327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:24.172460 (+ 133us) server_negotiation.cc:407] Connection header received
0926 07:19:24.172697 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:24.172701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:24.172757 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:24.172832 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:24.173131 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.173676 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.174806 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.175224 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.175353 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.175409 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:24.175586 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 07:19:24.175653 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:19:24.175801 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.171992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:34337 (local address 127.28.235.194:35283)
0926 07:19:24.172358 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:24.172372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:24.172525 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:24.172850 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:24.172853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:24.172870 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:24.172994 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.173000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.173800 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.173804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.174643 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.174672 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.175351 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.175355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.175455 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.175478 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:19:24.175569 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:19:24.175650 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":85,"threads_started":1}
I20260926 07:19:24.176085 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.172087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:40965 (local address 127.28.235.193:38217)
0926 07:19:24.172227 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:24.172232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:24.172598 (+ 366us) server_negotiation.cc:407] Connection header received
0926 07:19:24.172842 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:24.172844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:24.172894 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:24.173161 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:24.173437 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.174014 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.175092 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.175490 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.175630 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.175691 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:24.175880 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:19:24.175958 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:19:24.176095 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.172047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38217 (local address 127.28.235.194:40965)
0926 07:19:24.172489 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:24.172502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:24.172637 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:24.173171 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:24.173174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:24.173192 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:24.173309 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.173315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.174156 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.174162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.174952 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.174961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.175615 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.175619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.175725 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.175757 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:19:24.175851 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:19:24.175928 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":92,"threads_started":1}
I20260926 07:19:24.176486 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593e4ea5bcf34d3faf75ede7bb736674" candidate_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" is_pre_election: true
I20260926 07:19:24.176666 31778 raft_consensus.cc:2454] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e0765e2f434df0b3e79e53f2c0e62a in term 0.
I20260926 07:19:24.176719 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593e4ea5bcf34d3faf75ede7bb736674" candidate_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" is_pre_election: true
I20260926 07:19:24.177066 31499 raft_consensus.cc:2454] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e0765e2f434df0b3e79e53f2c0e62a in term 0.
I20260926 07:19:24.177114 31570 leader_election.cc:302] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1f80d1526ce45518d8a7886a9dcc49e, f0e0765e2f434df0b3e79e53f2c0e62a; no voters:
I20260926 07:19:24.177361 31856 raft_consensus.cc:2781] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:24.177430 31856 raft_consensus.cc:482] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:24.177479 31856 raft_consensus.cc:3037] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:24.178598 31856 raft_consensus.cc:504] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.179001 31856 leader_election.cc:288] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [CANDIDATE]: Term 1 election: Requested vote from peers 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193:38217), e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:24.179370 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593e4ea5bcf34d3faf75ede7bb736674" candidate_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f80d1526ce45518d8a7886a9dcc49e"
I20260926 07:19:24.179502 31778 raft_consensus.cc:3037] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:24.180469 31778 raft_consensus.cc:2454] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e0765e2f434df0b3e79e53f2c0e62a in term 1.
I20260926 07:19:24.180713 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593e4ea5bcf34d3faf75ede7bb736674" candidate_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd"
I20260926 07:19:24.180742 31570 leader_election.cc:302] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1f80d1526ce45518d8a7886a9dcc49e, f0e0765e2f434df0b3e79e53f2c0e62a; no voters:
I20260926 07:19:24.180843 31499 raft_consensus.cc:3037] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:24.180928 31856 raft_consensus.cc:2781] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:24.181159 31856 raft_consensus.cc:686] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 LEADER]: Becoming Leader. State: Replica: f0e0765e2f434df0b3e79e53f2c0e62a, State: Running, Role: LEADER
I20260926 07:19:24.181545 31856 consensus_queue.cc:237] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.181975 31499 raft_consensus.cc:2454] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e0765e2f434df0b3e79e53f2c0e62a in term 1.
I20260926 07:19:24.183609 31872 tablet_replica.cc:439] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: TxnStatusTablet state changed. Reason: New leader f0e0765e2f434df0b3e79e53f2c0e62a. Latest consensus state: current_term: 1 leader_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } } }
I20260926 07:19:24.183727 31872 tablet_replica.cc:442] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:19:24.184008 31874 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:19:24.184234 31874 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:19:24.184757 31339 catalog_manager.cc:5495] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a reported cstate change: term changed from 0 to 1, leader changed from <none> to f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194). New cstate: current_term: 1 leader_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:24.227597 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.222893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:37263 (local address 127.0.0.1:37702)
0926 07:19:24.223066 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:24.223085 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:24.223196 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:24.223607 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:24.223610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:24.223617 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:19:24.223727 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.223732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.224803 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.224806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.225550 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.225560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.226181 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.226184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.226298 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.226354 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:19:24.227287 (+ 933us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:19:24.227305 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:19:24.227374 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:19:24.227470 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 07:19:24.227622 31844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.223037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.28.235.194:37263)
0926 07:19:24.223164 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:24.223168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:24.223184 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:19:24.223352 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:24.223355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:24.223426 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:24.223555 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:24.223841 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.224647 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.225677 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.226037 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.226147 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.226546 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:24.227165 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:24.227293 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:24.227406 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 07:19:24.227472 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:19:24.233949 31499 raft_consensus.cc:1261] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Refusing update from remote peer f0e0765e2f434df0b3e79e53f2c0e62a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:24.234035 31778 raft_consensus.cc:1261] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Refusing update from remote peer f0e0765e2f434df0b3e79e53f2c0e62a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:24.234652 31872 consensus_queue.cc:1035] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:24.234869 31856 consensus_queue.cc:1035] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:24.238262 31854 tablet_replica.cc:439] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: TxnStatusTablet state changed. Reason: New leader f0e0765e2f434df0b3e79e53f2c0e62a. Latest consensus state: current_term: 1 leader_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } } }
I20260926 07:19:24.238391 31854 tablet_replica.cc:442] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:24.239524 31855 tablet_replica.cc:439] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: TxnStatusTablet state changed. Reason: New leader f0e0765e2f434df0b3e79e53f2c0e62a. Latest consensus state: current_term: 1 leader_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } } }
I20260926 07:19:24.239655 31855 tablet_replica.cc:442] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:24.242465 31872 tablet_replica.cc:439] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } } }
I20260926 07:19:24.242612 31872 tablet_replica.cc:442] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:19:24.242831 31879 mvcc.cc:205] Tried to move back new op lower bound from 7333507744694931456 to 7333507744490659840. Current Snapshot: MvccSnapshot[applied={T|T < 7333507744694931456}]
I20260926 07:19:24.243156 31854 tablet_replica.cc:439] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } } }
I20260926 07:19:24.243270 31854 tablet_replica.cc:442] T 593e4ea5bcf34d3faf75ede7bb736674 P 1e2f3a02f94a4dbab3e5a06b392f7efd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:24.243498 31872 tablet_replica.cc:439] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } } }
I20260926 07:19:24.243594 31872 tablet_replica.cc:442] T 593e4ea5bcf34d3faf75ede7bb736674 P f0e0765e2f434df0b3e79e53f2c0e62a: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:19:24.244445 31877 mvcc.cc:205] Tried to move back new op lower bound from 7333507744694931456 to 7333507744490659840. Current Snapshot: MvccSnapshot[applied={T|T < 7333507744694931456}]
I20260926 07:19:24.244822 31855 tablet_replica.cc:439] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } } }
I20260926 07:19:24.244941 31855 tablet_replica.cc:442] T 593e4ea5bcf34d3faf75ede7bb736674 P e1f80d1526ce45518d8a7886a9dcc49e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:24.245877 31878 mvcc.cc:205] Tried to move back new op lower bound from 7333507744694931456 to 7333507744490659840. Current Snapshot: MvccSnapshot[applied={T|T < 7333507744694931456}]
I20260926 07:19:24.256651 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.253107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.28.235.193:38217)
0926 07:19:24.253229 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:24.253234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:24.253248 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:19:24.253404 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:24.253407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:24.253465 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:24.253571 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:24.253813 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.254335 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.255191 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.255415 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.255527 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.255810 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:24.256250 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:24.256403 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:24.256437 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 07:19:24.256496 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:19:24.268399 31844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.264955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:44189 (local address 127.28.235.194:37263)
0926 07:19:24.265139 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:24.265144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:24.265258 (+ 114us) server_negotiation.cc:407] Connection header received
0926 07:19:24.265423 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:24.265427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:24.265481 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:24.265559 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:24.265824 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.266346 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.267275 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.267500 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.267608 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.267930 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:24.268009 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:24.268085 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:24.268194 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 07:19:24.268249 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 07:19:24.268421 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.264827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:37263 (local address 127.28.235.193:44189)
0926 07:19:24.265167 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:24.265187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:24.265319 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:24.265573 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:24.265575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:24.265583 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:19:24.265723 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.265729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.266492 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.266496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.267149 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.267162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.267614 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.267617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.267723 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.267769 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:19:24.268091 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:19:24.268100 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:19:24.268181 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:19:24.268265 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":90,"threads_started":1}
I20260926 07:19:24.275784 31778 raft_consensus.cc:1261] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Refusing update from remote peer 1e2f3a02f94a4dbab3e5a06b392f7efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:24.275946 31639 raft_consensus.cc:1261] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Refusing update from remote peer 1e2f3a02f94a4dbab3e5a06b392f7efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:24.276307 31859 consensus_queue.cc:1035] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:24.276712 31859 consensus_queue.cc:1035] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:24.299391 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.295396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:33349 (local address 127.28.235.193:38217)
0926 07:19:24.295531 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:24.295536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:24.295759 (+ 223us) server_negotiation.cc:407] Connection header received
0926 07:19:24.295899 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:24.295901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:24.295952 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:24.296036 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:24.296335 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.296910 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.298266 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.298535 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.298632 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.298939 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:24.299024 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:24.299105 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:24.299192 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 07:19:24.299247 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:19:24.299423 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:24.295322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38217 (local address 127.28.235.194:33349)
0926 07:19:24.295668 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:24.295686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:24.295798 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:24.296056 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:24.296059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:24.296067 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:19:24.296183 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.296189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.297050 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.297054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.298125 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:24.298138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:24.298660 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:24.298664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:24.298761 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:24.298804 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:19:24.299110 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:19:24.299119 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:19:24.299183 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:19:24.299272 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":126,"threads_started":1}
I20260926 07:19:24.326696 31340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:24.326931 31340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:24.330287 31340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c: Sending DeleteTablet for 3 replicas of tablet da33f64a04d641b29e143a512b957992
I20260926 07:19:24.331324 31479 tablet_service.cc:1514] Processing DeleteTablet for tablet da33f64a04d641b29e143a512b957992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:24 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:48658
I20260926 07:19:24.331398 31759 tablet_service.cc:1514] Processing DeleteTablet for tablet da33f64a04d641b29e143a512b957992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:24 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:38974
I20260926 07:19:24.331925 31898 tablet_replica.cc:330] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: stopping tablet replica
I20260926 07:19:24.332906 31619 tablet_service.cc:1514] Processing DeleteTablet for tablet da33f64a04d641b29e143a512b957992 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:24 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:37700
I20260926 07:19:24.333096 31340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:19:24.333485 31898 raft_consensus.cc:2229] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 07:19:24.333588 31340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:24.333768 31897 tablet_replica.cc:330] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: stopping tablet replica
I20260926 07:19:24.333767 31898 raft_consensus.cc:2258] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:24.334071 31897 raft_consensus.cc:2229] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:24.334527 31897 raft_consensus.cc:2258] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:24.335142 31899 tablet_replica.cc:330] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: stopping tablet replica
I20260926 07:19:24.336782 31897 ts_tablet_manager.cc:1905] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:24.340147 31897 ts_tablet_manager.cc:1918] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:24.340279 31619 tablet_service.cc:1467] Processing CreateTablet for tablet eb535d83334547d695f437dd78889c9c (DEFAULT_TABLE table=test-table [id=286902e036514c6381aabd5e2abe5a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:24.340340 31899 raft_consensus.cc:2229] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:24.340556 31899 raft_consensus.cc:2258] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:24.340608 31619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb535d83334547d695f437dd78889c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:24.342417 31899 ts_tablet_manager.cc:1905] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:24.342403 31898 ts_tablet_manager.cc:1905] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:24.343495 31759 tablet_service.cc:1467] Processing CreateTablet for tablet eb535d83334547d695f437dd78889c9c (DEFAULT_TABLE table=test-table [id=286902e036514c6381aabd5e2abe5a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:24.343770 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb535d83334547d695f437dd78889c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:24.345307 31899 ts_tablet_manager.cc:1918] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:24.345410 31899 log.cc:1196] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/da33f64a04d641b29e143a512b957992
I20260926 07:19:24.345510 31898 ts_tablet_manager.cc:1918] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:24.345593 31898 log.cc:1196] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/da33f64a04d641b29e143a512b957992
I20260926 07:19:24.345829 31897 log.cc:1196] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/da33f64a04d641b29e143a512b957992
I20260926 07:19:24.345898 31898 ts_tablet_manager.cc:1939] T da33f64a04d641b29e143a512b957992 P e1f80d1526ce45518d8a7886a9dcc49e: Deleting consensus metadata
I20260926 07:19:24.346166 31897 ts_tablet_manager.cc:1939] T da33f64a04d641b29e143a512b957992 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Deleting consensus metadata
I20260926 07:19:24.346374 31850 tablet_bootstrap.cc:492] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e: Bootstrap starting.
I20260926 07:19:24.347339 31850 tablet_bootstrap.cc:654] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:24.347548 31899 ts_tablet_manager.cc:1939] T da33f64a04d641b29e143a512b957992 P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting consensus metadata
I20260926 07:19:24.348726 31329 catalog_manager.cc:4841] TS f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263): tablet da33f64a04d641b29e143a512b957992 (table test-table [id=3c4f443db0414e709c37127022181aed]) successfully deleted
I20260926 07:19:24.348961 31327 catalog_manager.cc:4841] TS 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193:38217): tablet da33f64a04d641b29e143a512b957992 (table test-table [id=3c4f443db0414e709c37127022181aed]) successfully deleted
I20260926 07:19:24.349402 31326 catalog_manager.cc:4841] TS e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337): tablet da33f64a04d641b29e143a512b957992 (table test-table [id=3c4f443db0414e709c37127022181aed]) successfully deleted
I20260926 07:19:24.349625 31479 tablet_service.cc:1467] Processing CreateTablet for tablet eb535d83334547d695f437dd78889c9c (DEFAULT_TABLE table=test-table [id=286902e036514c6381aabd5e2abe5a45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:24.349897 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb535d83334547d695f437dd78889c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:24.352268 31848 tablet_bootstrap.cc:492] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd: Bootstrap starting.
I20260926 07:19:24.353255 31848 tablet_bootstrap.cc:654] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:24.353796 31849 tablet_bootstrap.cc:492] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a: Bootstrap starting.
I20260926 07:19:24.354722 31849 tablet_bootstrap.cc:654] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:24.356410 31850 tablet_bootstrap.cc:492] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e: No bootstrap required, opened a new log
I20260926 07:19:24.356500 31850 ts_tablet_manager.cc:1397] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20260926 07:19:24.356559 31849 tablet_bootstrap.cc:492] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a: No bootstrap required, opened a new log
I20260926 07:19:24.356632 31849 ts_tablet_manager.cc:1397] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:24.356992 31850 raft_consensus.cc:348] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.357095 31849 raft_consensus.cc:348] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.357183 31850 raft_consensus.cc:374] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:24.357204 31849 raft_consensus.cc:374] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:24.357234 31849 raft_consensus.cc:729] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e0765e2f434df0b3e79e53f2c0e62a, State: Initialized, Role: FOLLOWER
I20260926 07:19:24.357227 31850 raft_consensus.cc:729] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f80d1526ce45518d8a7886a9dcc49e, State: Initialized, Role: FOLLOWER
I20260926 07:19:24.357354 31849 consensus_queue.cc:260] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.357374 31850 consensus_queue.cc:260] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.357610 31849 ts_tablet_manager.cc:1428] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:24.357633 31850 ts_tablet_manager.cc:1428] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:19:24.360386 31848 tablet_bootstrap.cc:492] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd: No bootstrap required, opened a new log
I20260926 07:19:24.360484 31848 ts_tablet_manager.cc:1397] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260926 07:19:24.360909 31848 raft_consensus.cc:348] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.361059 31848 raft_consensus.cc:374] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:24.361094 31848 raft_consensus.cc:729] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2f3a02f94a4dbab3e5a06b392f7efd, State: Initialized, Role: FOLLOWER
I20260926 07:19:24.361217 31848 consensus_queue.cc:260] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.361451 31848 ts_tablet_manager.cc:1428] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:24.453914 31894 raft_consensus.cc:482] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:24.454116 31894 raft_consensus.cc:504] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.454650 31894 leader_election.cc:288] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263), e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:24.455157 31639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb535d83334547d695f437dd78889c9c" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" is_pre_election: true
I20260926 07:19:24.455171 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb535d83334547d695f437dd78889c9c" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" is_pre_election: true
I20260926 07:19:24.455317 31639 raft_consensus.cc:2454] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 0.
I20260926 07:19:24.455343 31779 raft_consensus.cc:2454] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 0.
I20260926 07:19:24.455639 31433 leader_election.cc:302] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f3a02f94a4dbab3e5a06b392f7efd, f0e0765e2f434df0b3e79e53f2c0e62a; no voters:
I20260926 07:19:24.455822 31894 raft_consensus.cc:2781] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:24.455888 31894 raft_consensus.cc:482] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:24.455926 31894 raft_consensus.cc:3037] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:24.457003 31894 raft_consensus.cc:504] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.457460 31894 leader_election.cc:288] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 election: Requested vote from peers f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263), e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:24.457811 31639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb535d83334547d695f437dd78889c9c" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a"
I20260926 07:19:24.457983 31639 raft_consensus.cc:3037] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:24.457980 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb535d83334547d695f437dd78889c9c" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f80d1526ce45518d8a7886a9dcc49e"
I20260926 07:19:24.458105 31779 raft_consensus.cc:3037] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:24.459194 31779 raft_consensus.cc:2454] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 1.
I20260926 07:19:24.459317 31639 raft_consensus.cc:2454] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 1.
I20260926 07:19:24.459532 31430 leader_election.cc:302] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f3a02f94a4dbab3e5a06b392f7efd, e1f80d1526ce45518d8a7886a9dcc49e; no voters:
I20260926 07:19:24.459738 31894 raft_consensus.cc:2781] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:24.459856 31894 raft_consensus.cc:686] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 LEADER]: Becoming Leader. State: Replica: 1e2f3a02f94a4dbab3e5a06b392f7efd, State: Running, Role: LEADER
I20260926 07:19:24.460230 31894 consensus_queue.cc:237] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:24.462718 31340 catalog_manager.cc:5495] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193). New cstate: current_term: 1 leader_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:24.497962 31639 raft_consensus.cc:1261] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Refusing update from remote peer 1e2f3a02f94a4dbab3e5a06b392f7efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:24.498054 31779 raft_consensus.cc:1261] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Refusing update from remote peer 1e2f3a02f94a4dbab3e5a06b392f7efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:24.498448 31859 consensus_queue.cc:1035] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:24.498843 31859 consensus_queue.cc:1035] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:19:24.512920 31879 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:19:26.064118 31340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:26.064270 31340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:26.067021 31340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c: Sending DeleteTablet for 3 replicas of tablet eb535d83334547d695f437dd78889c9c
I20260926 07:19:26.067651 31479 tablet_service.cc:1514] Processing DeleteTablet for tablet eb535d83334547d695f437dd78889c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:26 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:48658
I20260926 07:19:26.067770 31618 tablet_service.cc:1514] Processing DeleteTablet for tablet eb535d83334547d695f437dd78889c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:26 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:37700
I20260926 07:19:26.068003 31759 tablet_service.cc:1514] Processing DeleteTablet for tablet eb535d83334547d695f437dd78889c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:26 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:38974
I20260926 07:19:26.068100 31929 tablet_replica.cc:330] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd: stopping tablet replica
I20260926 07:19:26.068193 31930 tablet_replica.cc:330] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a: stopping tablet replica
I20260926 07:19:26.068271 31929 raft_consensus.cc:2229] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:26.068364 31930 raft_consensus.cc:2229] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:26.068547 31929 raft_consensus.cc:2258] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:26.068596 31930 raft_consensus.cc:2258] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:26.069298 31931 tablet_replica.cc:330] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e: stopping tablet replica
I20260926 07:19:26.069388 31929 ts_tablet_manager.cc:1905] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:26.069526 31931 raft_consensus.cc:2229] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:26.069674 31930 ts_tablet_manager.cc:1905] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:26.069809 31931 raft_consensus.cc:2258] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:26.070626 31931 ts_tablet_manager.cc:1905] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:26.072765 31929 ts_tablet_manager.cc:1918] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:19:26.072917 31929 log.cc:1196] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/eb535d83334547d695f437dd78889c9c
I20260926 07:19:26.072899 31930 ts_tablet_manager.cc:1918] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:19:26.073001 31930 log.cc:1196] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/eb535d83334547d695f437dd78889c9c
I20260926 07:19:26.073326 31929 ts_tablet_manager.cc:1939] T eb535d83334547d695f437dd78889c9c P 1e2f3a02f94a4dbab3e5a06b392f7efd: Deleting consensus metadata
I20260926 07:19:26.073369 31930 ts_tablet_manager.cc:1939] T eb535d83334547d695f437dd78889c9c P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting consensus metadata
I20260926 07:19:26.074270 31327 catalog_manager.cc:4841] TS 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193:38217): tablet eb535d83334547d695f437dd78889c9c (table test-table [id=286902e036514c6381aabd5e2abe5a45]) successfully deleted
I20260926 07:19:26.074363 31329 catalog_manager.cc:4841] TS f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263): tablet eb535d83334547d695f437dd78889c9c (table test-table [id=286902e036514c6381aabd5e2abe5a45]) successfully deleted
I20260926 07:19:26.074787 31931 ts_tablet_manager.cc:1918] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:19:26.074880 31931 log.cc:1196] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/eb535d83334547d695f437dd78889c9c
I20260926 07:19:26.075140 31931 ts_tablet_manager.cc:1939] T eb535d83334547d695f437dd78889c9c P e1f80d1526ce45518d8a7886a9dcc49e: Deleting consensus metadata
I20260926 07:19:26.075909 31326 catalog_manager.cc:4841] TS e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337): tablet eb535d83334547d695f437dd78889c9c (table test-table [id=286902e036514c6381aabd5e2abe5a45]) successfully deleted
Sep 26 07:19:26 dist-test-slave-7zlw krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407166, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:19:26 dist-test-slave-7zlw krb5kdc[31297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407166, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:26.107146 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:26.089344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.28.235.254:34877)
0926 07:19:26.089637 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:26.089645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:26.089664 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:19:26.089784 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:26.089788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:26.089841 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:26.089940 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:26.090722 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:26.091368 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:26.092298 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:26.092531 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:26.092618 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:26.103174 (+ 10556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:26.103179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:26.103181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:26.103222 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:26.105279 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:26.105792 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:26.105795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:26.105797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:26.105888 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:26.106211 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:26.106214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:26.106215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:26.106419 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:26.106532 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:26.106785 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 07:19:26.106902 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":111,"threads_started":1}
I20260926 07:19:26.110213 31340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:19:26.110736 31340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:26.115432 31479 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d (DEFAULT_TABLE table=test-table [id=a4d75ab0e1a2448bb62329e0b5f7291d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:26.115481 31618 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d (DEFAULT_TABLE table=test-table [id=a4d75ab0e1a2448bb62329e0b5f7291d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:26.115621 31759 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d (DEFAULT_TABLE table=test-table [id=a4d75ab0e1a2448bb62329e0b5f7291d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:26.115754 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:26.115761 31618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:26.115926 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:26.117973 31940 tablet_bootstrap.cc:492] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e: Bootstrap starting.
I20260926 07:19:26.118542 31941 tablet_bootstrap.cc:492] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a: Bootstrap starting.
I20260926 07:19:26.118780 31942 tablet_bootstrap.cc:492] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd: Bootstrap starting.
I20260926 07:19:26.119369 31940 tablet_bootstrap.cc:654] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:26.119503 31941 tablet_bootstrap.cc:654] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:26.119678 31942 tablet_bootstrap.cc:654] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:26.120754 31941 tablet_bootstrap.cc:492] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a: No bootstrap required, opened a new log
I20260926 07:19:26.120754 31940 tablet_bootstrap.cc:492] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e: No bootstrap required, opened a new log
I20260926 07:19:26.120841 31940 ts_tablet_manager.cc:1397] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:26.120841 31941 ts_tablet_manager.cc:1397] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:19:26.121075 31942 tablet_bootstrap.cc:492] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd: No bootstrap required, opened a new log
I20260926 07:19:26.121168 31942 ts_tablet_manager.cc:1397] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:19:26.121310 31941 raft_consensus.cc:348] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.121310 31940 raft_consensus.cc:348] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.121421 31940 raft_consensus.cc:374] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:26.121421 31941 raft_consensus.cc:374] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:26.121455 31940 raft_consensus.cc:729] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f80d1526ce45518d8a7886a9dcc49e, State: Initialized, Role: FOLLOWER
I20260926 07:19:26.121455 31941 raft_consensus.cc:729] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e0765e2f434df0b3e79e53f2c0e62a, State: Initialized, Role: FOLLOWER
I20260926 07:19:26.121531 31942 raft_consensus.cc:348] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.121589 31941 consensus_queue.cc:260] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.121589 31940 consensus_queue.cc:260] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.121670 31942 raft_consensus.cc:374] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:26.121696 31942 raft_consensus.cc:729] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2f3a02f94a4dbab3e5a06b392f7efd, State: Initialized, Role: FOLLOWER
I20260926 07:19:26.121882 31941 ts_tablet_manager.cc:1428] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:26.121869 31942 consensus_queue.cc:260] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.121944 31940 ts_tablet_manager.cc:1428] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:26.123090 31942 ts_tablet_manager.cc:1428] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:19:26.182454 31906 raft_consensus.cc:482] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:26.182631 31906 raft_consensus.cc:504] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.183029 31906 leader_election.cc:288] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263), e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:26.183456 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc9aee151f74ad9b3ec4a8f36f44b2d" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" is_pre_election: true
I20260926 07:19:26.183445 31639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc9aee151f74ad9b3ec4a8f36f44b2d" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" is_pre_election: true
I20260926 07:19:26.183590 31778 raft_consensus.cc:2454] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 0.
I20260926 07:19:26.183624 31639 raft_consensus.cc:2454] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 0.
I20260926 07:19:26.183949 31430 leader_election.cc:302] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f3a02f94a4dbab3e5a06b392f7efd, e1f80d1526ce45518d8a7886a9dcc49e; no voters:
I20260926 07:19:26.184135 31906 raft_consensus.cc:2781] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:26.184204 31906 raft_consensus.cc:482] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:26.184231 31906 raft_consensus.cc:3037] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:26.185388 31906 raft_consensus.cc:504] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.185750 31906 leader_election.cc:288] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 election: Requested vote from peers f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263), e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:26.186156 31639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc9aee151f74ad9b3ec4a8f36f44b2d" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a"
I20260926 07:19:26.186314 31639 raft_consensus.cc:3037] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:26.186318 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc9aee151f74ad9b3ec4a8f36f44b2d" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f80d1526ce45518d8a7886a9dcc49e"
I20260926 07:19:26.186509 31778 raft_consensus.cc:3037] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:26.187376 31639 raft_consensus.cc:2454] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 1.
I20260926 07:19:26.187669 31778 raft_consensus.cc:2454] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 1.
I20260926 07:19:26.187698 31433 leader_election.cc:302] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f3a02f94a4dbab3e5a06b392f7efd, f0e0765e2f434df0b3e79e53f2c0e62a; no voters:
I20260926 07:19:26.187914 31906 raft_consensus.cc:2781] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:26.188048 31906 raft_consensus.cc:686] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 LEADER]: Becoming Leader. State: Replica: 1e2f3a02f94a4dbab3e5a06b392f7efd, State: Running, Role: LEADER
I20260926 07:19:26.188241 31906 consensus_queue.cc:237] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.190234 31340 catalog_manager.cc:5495] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193). New cstate: current_term: 1 leader_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:26.211083 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:26.207712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.28.235.193:38217)
0926 07:19:26.208001 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:26.208008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:26.208018 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:19:26.208070 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:26.208073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:26.208137 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:26.208230 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:26.208510 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:26.209141 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:26.210025 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:26.210267 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:26.210379 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:26.210629 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:26.210706 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:26.210785 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:26.210873 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 07:19:26.210925 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":120,"threads_started":1}
I20260926 07:19:26.218034 31639 raft_consensus.cc:1261] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Refusing update from remote peer 1e2f3a02f94a4dbab3e5a06b392f7efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:26.218403 31778 raft_consensus.cc:1261] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Refusing update from remote peer 1e2f3a02f94a4dbab3e5a06b392f7efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:26.218510 31923 consensus_queue.cc:1035] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:26.218894 31906 consensus_queue.cc:1035] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:26.248848 31340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:26.249055 31340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:26.251305 31340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c: Sending DeleteTablet for 3 replicas of tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d
I20260926 07:19:26.251850 31479 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:26 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:48658
I20260926 07:19:26.251993 31618 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:26 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:37700
I20260926 07:19:26.252049 31929 tablet_replica.cc:330] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd: stopping tablet replica
I20260926 07:19:26.251987 31759 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:26 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:38974
I20260926 07:19:26.252158 31930 tablet_replica.cc:330] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a: stopping tablet replica
I20260926 07:19:26.252173 31929 raft_consensus.cc:2229] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:26.252259 31930 raft_consensus.cc:2229] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:26.252355 31929 raft_consensus.cc:2258] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:26.252363 31930 raft_consensus.cc:2258] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:26.252542 31931 tablet_replica.cc:330] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e: stopping tablet replica
I20260926 07:19:26.252704 31931 raft_consensus.cc:2229] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:26.252833 31931 raft_consensus.cc:2258] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:26.253080 31929 ts_tablet_manager.cc:1905] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:26.253173 31930 ts_tablet_manager.cc:1905] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:26.253532 31340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:19:26.253724 31931 ts_tablet_manager.cc:1905] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:19:26.254024 31340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:26.255642 31930 ts_tablet_manager.cc:1918] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:26.255642 31929 ts_tablet_manager.cc:1918] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:26.255712 31929 log.cc:1196] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/6fc9aee151f74ad9b3ec4a8f36f44b2d
I20260926 07:19:26.255712 31930 log.cc:1196] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/6fc9aee151f74ad9b3ec4a8f36f44b2d
I20260926 07:19:26.255966 31930 ts_tablet_manager.cc:1939] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting consensus metadata
I20260926 07:19:26.255966 31929 ts_tablet_manager.cc:1939] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P 1e2f3a02f94a4dbab3e5a06b392f7efd: Deleting consensus metadata
I20260926 07:19:26.256294 31931 ts_tablet_manager.cc:1918] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:26.256420 31931 log.cc:1196] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/6fc9aee151f74ad9b3ec4a8f36f44b2d
I20260926 07:19:26.256742 31931 ts_tablet_manager.cc:1939] T 6fc9aee151f74ad9b3ec4a8f36f44b2d P e1f80d1526ce45518d8a7886a9dcc49e: Deleting consensus metadata
I20260926 07:19:26.257071 31329 catalog_manager.cc:4841] TS f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263): tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d (table test-table [id=a4d75ab0e1a2448bb62329e0b5f7291d]) successfully deleted
I20260926 07:19:26.257154 31327 catalog_manager.cc:4841] TS 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193:38217): tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d (table test-table [id=a4d75ab0e1a2448bb62329e0b5f7291d]) successfully deleted
I20260926 07:19:26.257441 31326 catalog_manager.cc:4841] TS e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337): tablet 6fc9aee151f74ad9b3ec4a8f36f44b2d (table test-table [id=a4d75ab0e1a2448bb62329e0b5f7291d]) successfully deleted
I20260926 07:19:26.260133 31479 tablet_service.cc:1467] Processing CreateTablet for tablet 7c628bb5b1e74edcbfbb4127b81139f1 (DEFAULT_TABLE table=test-table [id=7d9d9937b98048ecbe1712e96c88fb49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:26.260234 31759 tablet_service.cc:1467] Processing CreateTablet for tablet 7c628bb5b1e74edcbfbb4127b81139f1 (DEFAULT_TABLE table=test-table [id=7d9d9937b98048ecbe1712e96c88fb49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:26.260463 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c628bb5b1e74edcbfbb4127b81139f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:26.260489 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c628bb5b1e74edcbfbb4127b81139f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:26.262450 31940 tablet_bootstrap.cc:492] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e: Bootstrap starting.
I20260926 07:19:26.262571 31942 tablet_bootstrap.cc:492] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Bootstrap starting.
I20260926 07:19:26.263135 31618 tablet_service.cc:1467] Processing CreateTablet for tablet 7c628bb5b1e74edcbfbb4127b81139f1 (DEFAULT_TABLE table=test-table [id=7d9d9937b98048ecbe1712e96c88fb49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:26.263373 31942 tablet_bootstrap.cc:654] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:26.263425 31618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c628bb5b1e74edcbfbb4127b81139f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:26.263567 31940 tablet_bootstrap.cc:654] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:26.264456 31942 tablet_bootstrap.cc:492] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd: No bootstrap required, opened a new log
I20260926 07:19:26.264540 31942 ts_tablet_manager.cc:1397] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:19:26.264964 31940 tablet_bootstrap.cc:492] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e: No bootstrap required, opened a new log
I20260926 07:19:26.264911 31942 raft_consensus.cc:348] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.265061 31942 raft_consensus.cc:374] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:26.265095 31940 ts_tablet_manager.cc:1397] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:26.265100 31942 raft_consensus.cc:729] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2f3a02f94a4dbab3e5a06b392f7efd, State: Initialized, Role: FOLLOWER
I20260926 07:19:26.265218 31942 consensus_queue.cc:260] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.265471 31942 ts_tablet_manager.cc:1428] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:26.265580 31940 raft_consensus.cc:348] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.265708 31940 raft_consensus.cc:374] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:26.265749 31940 raft_consensus.cc:729] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f80d1526ce45518d8a7886a9dcc49e, State: Initialized, Role: FOLLOWER
I20260926 07:19:26.265920 31940 consensus_queue.cc:260] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.266228 31940 ts_tablet_manager.cc:1428] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:26.266518 31941 tablet_bootstrap.cc:492] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a: Bootstrap starting.
I20260926 07:19:26.267488 31941 tablet_bootstrap.cc:654] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:26.268657 31941 tablet_bootstrap.cc:492] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a: No bootstrap required, opened a new log
I20260926 07:19:26.268738 31941 ts_tablet_manager.cc:1397] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:19:26.269109 31941 raft_consensus.cc:348] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.269213 31941 raft_consensus.cc:374] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:26.269236 31941 raft_consensus.cc:729] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e0765e2f434df0b3e79e53f2c0e62a, State: Initialized, Role: FOLLOWER
I20260926 07:19:26.269380 31941 consensus_queue.cc:260] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.269655 31941 ts_tablet_manager.cc:1428] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:26.364812 31923 raft_consensus.cc:482] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:26.364985 31923 raft_consensus.cc:504] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.365494 31923 leader_election.cc:288] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263), e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:26.365897 31639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c628bb5b1e74edcbfbb4127b81139f1" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" is_pre_election: true
I20260926 07:19:26.365952 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c628bb5b1e74edcbfbb4127b81139f1" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" is_pre_election: true
I20260926 07:19:26.366027 31639 raft_consensus.cc:2454] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 0.
I20260926 07:19:26.366117 31778 raft_consensus.cc:2454] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 0.
I20260926 07:19:26.366333 31433 leader_election.cc:302] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f3a02f94a4dbab3e5a06b392f7efd, f0e0765e2f434df0b3e79e53f2c0e62a; no voters:
I20260926 07:19:26.366566 31923 raft_consensus.cc:2781] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:26.366624 31923 raft_consensus.cc:482] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:26.366652 31923 raft_consensus.cc:3037] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:26.367738 31923 raft_consensus.cc:504] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.368175 31923 leader_election.cc:288] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 election: Requested vote from peers f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263), e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337)
I20260926 07:19:26.368539 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c628bb5b1e74edcbfbb4127b81139f1" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f80d1526ce45518d8a7886a9dcc49e"
I20260926 07:19:26.368506 31639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c628bb5b1e74edcbfbb4127b81139f1" candidate_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a"
I20260926 07:19:26.368681 31778 raft_consensus.cc:3037] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:26.368710 31639 raft_consensus.cc:3037] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:26.369783 31778 raft_consensus.cc:2454] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 1.
I20260926 07:19:26.369814 31639 raft_consensus.cc:2454] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2f3a02f94a4dbab3e5a06b392f7efd in term 1.
I20260926 07:19:26.370079 31430 leader_election.cc:302] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2f3a02f94a4dbab3e5a06b392f7efd, e1f80d1526ce45518d8a7886a9dcc49e; no voters:
I20260926 07:19:26.370260 31923 raft_consensus.cc:2781] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:26.370352 31923 raft_consensus.cc:686] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [term 1 LEADER]: Becoming Leader. State: Replica: 1e2f3a02f94a4dbab3e5a06b392f7efd, State: Running, Role: LEADER
I20260926 07:19:26.370499 31923 consensus_queue.cc:237] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } }
I20260926 07:19:26.372155 31340 catalog_manager.cc:5495] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193). New cstate: current_term: 1 leader_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2f3a02f94a4dbab3e5a06b392f7efd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:26.400991 31779 raft_consensus.cc:1261] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Refusing update from remote peer 1e2f3a02f94a4dbab3e5a06b392f7efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:26.401224 31639 raft_consensus.cc:1261] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Refusing update from remote peer 1e2f3a02f94a4dbab3e5a06b392f7efd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:26.401580 31923 consensus_queue.cc:1035] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f80d1526ce45518d8a7886a9dcc49e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 34337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:26.401705 31906 consensus_queue.cc:1035] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e0765e2f434df0b3e79e53f2c0e62a" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:19:26.415987 31912 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:19:27.982555 31341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:27.982731 31341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:27.985375 31341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53bfeee7da2541988ef3376d3bd0417c: Sending DeleteTablet for 3 replicas of tablet 7c628bb5b1e74edcbfbb4127b81139f1
I20260926 07:19:27.985930 31479 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c628bb5b1e74edcbfbb4127b81139f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:27 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:48658
I20260926 07:19:27.985998 31759 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c628bb5b1e74edcbfbb4127b81139f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:27 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:38974
I20260926 07:19:27.986279 31618 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c628bb5b1e74edcbfbb4127b81139f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:27 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:37700
I20260926 07:19:27.986384 31981 tablet_replica.cc:330] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e: stopping tablet replica
I20260926 07:19:27.986406 31980 tablet_replica.cc:330] T 7c628bb5b1e74edcbfbb4127b81139f1 P 1e2f3a02f94a4dbab3e5a06b392f7efd: stopping tablet replica
I20260926 07:19:27.986471 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 31414
I20260926 07:19:27.986536 31981 raft_consensus.cc:2229] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:27.987287 31981 raft_consensus.cc:2258] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:27.987354 31982 tablet_replica.cc:330] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a: stopping tablet replica
I20260926 07:19:27.987684 31982 raft_consensus.cc:2229] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:27.987838 31982 raft_consensus.cc:2258] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:27.988046 31981 ts_tablet_manager.cc:1905] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:27.989514 31982 ts_tablet_manager.cc:1905] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:27.990636 31981 ts_tablet_manager.cc:1918] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:19:27.990728 31981 log.cc:1196] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/7c628bb5b1e74edcbfbb4127b81139f1
I20260926 07:19:27.990990 31981 ts_tablet_manager.cc:1939] T 7c628bb5b1e74edcbfbb4127b81139f1 P e1f80d1526ce45518d8a7886a9dcc49e: Deleting consensus metadata
I20260926 07:19:27.991668 31326 catalog_manager.cc:4841] TS e1f80d1526ce45518d8a7886a9dcc49e (127.28.235.195:34337): tablet 7c628bb5b1e74edcbfbb4127b81139f1 (table test-table [id=7d9d9937b98048ecbe1712e96c88fb49]) successfully deleted
I20260926 07:19:27.992841 31982 ts_tablet_manager.cc:1918] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:19:27.992930 31982 log.cc:1196] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/7c628bb5b1e74edcbfbb4127b81139f1
I20260926 07:19:27.993235 31982 ts_tablet_manager.cc:1939] T 7c628bb5b1e74edcbfbb4127b81139f1 P f0e0765e2f434df0b3e79e53f2c0e62a: Deleting consensus metadata
I20260926 07:19:27.994081 31329 catalog_manager.cc:4841] TS f0e0765e2f434df0b3e79e53f2c0e62a (127.28.235.194:37263): tablet 7c628bb5b1e74edcbfbb4127b81139f1 (table test-table [id=7d9d9937b98048ecbe1712e96c88fb49]) successfully deleted
I20260926 07:19:27.996106 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 31554
W20260926 07:19:27.996109 31327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:38217 (error 108)
I20260926 07:19:27.998723 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:27.998264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38217 (local address 127.0.0.1:48698)
0926 07:19:27.998500 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:27.998577 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:38217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
W20260926 07:19:27.998901 31327 catalog_manager.cc:4584] TS 1e2f3a02f94a4dbab3e5a06b392f7efd (127.28.235.193:38217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c628bb5b1e74edcbfbb4127b81139f1: Network error: Client connection negotiation failed: client connection to 127.28.235.193:38217: connect: Connection refused (error 111)
I20260926 07:19:28.003475 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 31694
I20260926 07:19:28.009759 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 31310
2026-09-26T07:19:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6333 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:28 dist-test-slave-7zlw krb5kdc[31987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:28 dist-test-slave-7zlw krb5kdc[31987](info): set up 2 sockets
Sep 26 07:19:28 dist-test-slave-7zlw krb5kdc[31987](info): commencing operation
krb5kdc: starting...
W20260926 07:19:29.129745 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:29 dist-test-slave-7zlw krb5kdc[31987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:29.260128 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:46415
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:46415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:29.372350 32000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:29.372620 32000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:29.372660 32000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:29.376736 32000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:29.376796 32000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:29.376816 32000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:29.376834 32000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:29.376876 32000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:29.382270 32000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:46415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:46415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:29.383349 32000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:29.384441 32000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:29.393366 32008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:29.393519 32000 server_base.cc:1034] running on GCE node
W20260926 07:19:29.393497 32006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:29.393414 32005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:29.394213 32000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:29.395318 32000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:29.396562 32000 hybrid_clock.cc:648] HybridClock initialized: now 1790407169396522 us; error 51 us; skew 500 ppm
Sep 26 07:19:29 dist-test-slave-7zlw krb5kdc[31987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407169, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:29.399524 32000 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:29.400411 32000 webserver.cc:460] Webserver started at http://127.28.235.254:42095/ using document root <none> and password file <none>
I20260926 07:19:29.400918 32000 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:29.400975 32000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:29.401216 32000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:29.403182 32000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "ac6ce0bb24b04126a7fe446a5e89804d"
format_stamp: "Formatted at 2026-09-26 07:19:29 on dist-test-slave-7zlw"
server_key: "151c3245c143ddaf2d529a0fcfed9996"
server_key_iv: "d5e080605316195af5fafd854fd64b79"
server_key_version: "encryptionkey@0"
I20260926 07:19:29.403677 32000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "ac6ce0bb24b04126a7fe446a5e89804d"
format_stamp: "Formatted at 2026-09-26 07:19:29 on dist-test-slave-7zlw"
server_key: "151c3245c143ddaf2d529a0fcfed9996"
server_key_iv: "d5e080605316195af5fafd854fd64b79"
server_key_version: "encryptionkey@0"
I20260926 07:19:29.407531 32000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:19:29.410132 32015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:29.411477 32000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:19:29.411629 32000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "ac6ce0bb24b04126a7fe446a5e89804d"
format_stamp: "Formatted at 2026-09-26 07:19:29 on dist-test-slave-7zlw"
server_key: "151c3245c143ddaf2d529a0fcfed9996"
server_key_iv: "d5e080605316195af5fafd854fd64b79"
server_key_version: "encryptionkey@0"
I20260926 07:19:29.411741 32000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:29.428265 32000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:29.431579 32000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:29.431793 32000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:29.439129 32000 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:46415
I20260926 07:19:29.439165 32067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:46415 every 8 connection(s)
I20260926 07:19:29.440155 32000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:29.443181 32068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:29.446995 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 32000
I20260926 07:19:29.447090 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:29.447373 29615 external_mini_cluster.cc:1428] Setting key 3f36186feb69f7850778b025e5c7b3bc
I20260926 07:19:29.449720 32068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d: Bootstrap starting.
I20260926 07:19:29.452593 32068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:29.453550 32068 log.cc:824] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:29.455498 32068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d: No bootstrap required, opened a new log
Sep 26 07:19:29 dist-test-slave-7zlw krb5kdc[31987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:29.459453 32068 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ce0bb24b04126a7fe446a5e89804d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46415 } }
I20260926 07:19:29.459715 32068 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:29.459832 32068 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6ce0bb24b04126a7fe446a5e89804d, State: Initialized, Role: FOLLOWER
I20260926 07:19:29.460362 32068 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ce0bb24b04126a7fe446a5e89804d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46415 } }
I20260926 07:19:29.460529 32068 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:29.460596 32068 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:29.460688 32068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:29.461890 32068 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ce0bb24b04126a7fe446a5e89804d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46415 } }
I20260926 07:19:29.462261 32068 leader_election.cc:302] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac6ce0bb24b04126a7fe446a5e89804d; no voters:
I20260926 07:19:29.462641 32068 leader_election.cc:288] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:29.462741 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:29.448913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.28.235.254:46415)
0926 07:19:29.449421 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:29.449441 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:29.449484 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:19:29.450331 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:29.450356 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:29.450788 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:29.451197 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:29.451912 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:29.453460 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:29.454377 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:29.454871 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:29.455042 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:29.457503 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:29.457508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:29.457523 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:29.457550 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:29.459360 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:29.459829 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:29.459835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:29.459843 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:29.459994 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:29.460304 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:29.460307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:29.460309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:29.460855 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:29.461208 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:29.461378 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 07:19:29.461702 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":129,"threads_started":1}
I20260926 07:19:29.462764 32073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:29.463001 32073 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [term 1 LEADER]: Becoming Leader. State: Replica: ac6ce0bb24b04126a7fe446a5e89804d, State: Running, Role: LEADER
I20260926 07:19:29.463312 32073 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ce0bb24b04126a7fe446a5e89804d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46415 } }
I20260926 07:19:29.463981 32068 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:29.464505 32074 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac6ce0bb24b04126a7fe446a5e89804d. Latest consensus state: current_term: 1 leader_uuid: "ac6ce0bb24b04126a7fe446a5e89804d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ce0bb24b04126a7fe446a5e89804d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46415 } } }
I20260926 07:19:29.464630 32074 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:29.464934 32075 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac6ce0bb24b04126a7fe446a5e89804d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac6ce0bb24b04126a7fe446a5e89804d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46415 } } }
I20260926 07:19:29.465116 32075 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:29.465423 32077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:29.469185 32077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:29.475137 32077 catalog_manager.cc:1347] Generated new cluster ID: 3fb04779a2eb4e58b9b35e833414aef7
I20260926 07:19:29.475220 32077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:29.490566 32077 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:29.491603 32077 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:29.495743 32077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac6ce0bb24b04126a7fe446a5e89804d: Generated new TSK 0
I20260926 07:19:29.496480 32077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:19:29.529501 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46415
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:29.652160 32094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:29.652429 32094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:29.652473 32094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:29.656702 32094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:29.656766 32094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:29.656827 32094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:29.662236 32094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:29.663371 32094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:29.664497 32094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:29.674849 32100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:29.674842 32102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:29.674842 32099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:29.675318 32094 server_base.cc:1034] running on GCE node
I20260926 07:19:29.675727 32094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:29.676445 32094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:29.677608 32094 hybrid_clock.cc:648] HybridClock initialized: now 1790407169677590 us; error 33 us; skew 500 ppm
Sep 26 07:19:29 dist-test-slave-7zlw krb5kdc[31987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407169, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:29.680500 32094 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:19:29.681455 32094 webserver.cc:460] Webserver started at http://127.28.235.193:41163/ using document root <none> and password file <none>
I20260926 07:19:29.682018 32094 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:29.682129 32094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:29.682339 32094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:29.684545 32094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "29ee7b3547e54d09a6c70f50025d5200"
format_stamp: "Formatted at 2026-09-26 07:19:29 on dist-test-slave-7zlw"
server_key: "22ac74cd070e19e30ba9cf12ddd7570d"
server_key_iv: "350dad67940ec4de1bad89236826e640"
server_key_version: "encryptionkey@0"
I20260926 07:19:29.685149 32094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "29ee7b3547e54d09a6c70f50025d5200"
format_stamp: "Formatted at 2026-09-26 07:19:29 on dist-test-slave-7zlw"
server_key: "22ac74cd070e19e30ba9cf12ddd7570d"
server_key_iv: "350dad67940ec4de1bad89236826e640"
server_key_version: "encryptionkey@0"
I20260926 07:19:29.688760 32094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:29.691352 32109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:29.692678 32094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:19:29.692831 32094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "29ee7b3547e54d09a6c70f50025d5200"
format_stamp: "Formatted at 2026-09-26 07:19:29 on dist-test-slave-7zlw"
server_key: "22ac74cd070e19e30ba9cf12ddd7570d"
server_key_iv: "350dad67940ec4de1bad89236826e640"
server_key_version: "encryptionkey@0"
I20260926 07:19:29.692945 32094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:29.707626 32094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:29.711139 32094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:29.711369 32094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:29.711990 32094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:29.712836 32094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:29.712903 32094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:29.712975 32094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:29.713014 32094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:29.723372 32094 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:45777
I20260926 07:19:29.723392 32222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:45777 every 8 connection(s)
I20260926 07:19:29.724365 32094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:29.726969 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 32094
I20260926 07:19:29.727088 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:29.727459 29615 external_mini_cluster.cc:1428] Setting key 08865ee72d2433c92183e538f7fd7d27
Sep 26 07:19:29 dist-test-slave-7zlw krb5kdc[31987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407169, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:19:29.742761 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:29.726705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:33201 (local address 127.28.235.254:46415)
0926 07:19:29.726860 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:29.726864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:29.728132 (+ 1268us) server_negotiation.cc:407] Connection header received
0926 07:19:29.729758 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:29.729765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:29.729830 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:29.729938 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:29.731801 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:29.732397 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:29.733985 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:29.734247 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:29.734377 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:29.738208 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:29.738212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:29.738214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:29.738264 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:29.740096 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:29.740937 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:29.740947 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:29.740950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:29.741084 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:29.741526 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:29.741529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:29.741531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:29.741742 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:29.741855 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:29.742438 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 07:19:29.742587 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:19:29.743701 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:29.727182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:46415 (local address 127.28.235.193:33201)
0926 07:19:29.727829 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:29.727878 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:29.729501 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:29.730204 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:29.730212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:29.730714 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:29.731543 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:29.731569 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:29.732561 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:29.732565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:29.733822 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:29.733836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:29.734428 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:29.734433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:29.734629 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:29.735500 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:29.735530 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:29.737956 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:29.740384 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:29.740392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:29.740414 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:29.740811 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:29.741239 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:29.741245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:29.741247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:29.741417 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:29.741866 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:29.741876 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:29.742182 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 07:19:29.742384 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 07:19:29.742779 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":165,"threads_started":1}
I20260926 07:19:29.745434 32223 heartbeater.cc:347] Connected to a master server at 127.28.235.254:46415
I20260926 07:19:29.745759 32223 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:29.746332 32223 heartbeater.cc:511] Master 127.28.235.254:46415 requested a full tablet report, sending...
I20260926 07:19:29.748425 32030 ts_manager.cc:194] Registered new tserver with Master: 29ee7b3547e54d09a6c70f50025d5200 (127.28.235.193:45777)
I20260926 07:19:29.750265 32030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:33201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:19:29.762022 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46415
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:29.882138 32228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:29.882400 32228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:29.882444 32228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:29.886495 32228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:29.886557 32228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:29.886617 32228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:29.891937 32228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:29.893260 32228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:29.894315 32228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:29.904512 32233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:29.904512 32236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:29.904541 32234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:29.904806 32228 server_base.cc:1034] running on GCE node
I20260926 07:19:29.905405 32228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:29.906159 32228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:29.907342 32228 hybrid_clock.cc:648] HybridClock initialized: now 1790407169907325 us; error 31 us; skew 500 ppm
Sep 26 07:19:29 dist-test-slave-7zlw krb5kdc[31987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407169, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:29.910468 32228 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:19:29.911372 32228 webserver.cc:460] Webserver started at http://127.28.235.194:39671/ using document root <none> and password file <none>
I20260926 07:19:29.911903 32228 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:29.911958 32228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:29.912125 32228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:29.914664 32228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "f34e50688cc446b1abfce800672cb09e"
format_stamp: "Formatted at 2026-09-26 07:19:29 on dist-test-slave-7zlw"
server_key: "91f5349cf1ed7ce1e14665649b7e5447"
server_key_iv: "cafc397ef6cf53f186476949ff9d8bf4"
server_key_version: "encryptionkey@0"
I20260926 07:19:29.915292 32228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "f34e50688cc446b1abfce800672cb09e"
format_stamp: "Formatted at 2026-09-26 07:19:29 on dist-test-slave-7zlw"
server_key: "91f5349cf1ed7ce1e14665649b7e5447"
server_key_iv: "cafc397ef6cf53f186476949ff9d8bf4"
server_key_version: "encryptionkey@0"
I20260926 07:19:29.919612 32228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:19:29.922314 32243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:29.923697 32228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:19:29.923801 32228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "f34e50688cc446b1abfce800672cb09e"
format_stamp: "Formatted at 2026-09-26 07:19:29 on dist-test-slave-7zlw"
server_key: "91f5349cf1ed7ce1e14665649b7e5447"
server_key_iv: "cafc397ef6cf53f186476949ff9d8bf4"
server_key_version: "encryptionkey@0"
I20260926 07:19:29.923945 32228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:29.940706 32228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:29.944228 32228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:29.944458 32228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:29.945125 32228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:29.946012 32228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:29.946058 32228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:29.946097 32228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:29.946151 32228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:29.955295 32228 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:45601
I20260926 07:19:29.955314 32356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:45601 every 8 connection(s)
I20260926 07:19:29.956215 32228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:29.959739 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 32228
I20260926 07:19:29.959827 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:29.960110 29615 external_mini_cluster.cc:1428] Setting key bbdf1eb6dbc756cbcb6c4f4eb1547e6d
Sep 26 07:19:29 dist-test-slave-7zlw krb5kdc[31987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407169, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
I20260926 07:19:29.973749 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:29.958189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:35233 (local address 127.28.235.254:46415)
0926 07:19:29.958336 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:29.958341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:29.959280 (+ 939us) server_negotiation.cc:407] Connection header received
0926 07:19:29.960701 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:29.960704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:29.960759 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:29.960849 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:29.962278 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:29.963246 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:29.965316 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:29.965545 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:29.965655 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:29.968846 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:29.968851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:29.968854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:29.968889 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:29.970820 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:29.971565 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:29.971570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:29.971572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:29.971676 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:29.972165 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:29.972169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:29.972171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:29.972490 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:29.972607 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:29.973358 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 07:19:29.973524 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:19:29.974721 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:29.958644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:46415 (local address 127.28.235.194:35233)
0926 07:19:29.959132 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:29.959175 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:29.960420 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:29.961063 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:29.961072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:29.961646 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:29.962133 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:29.962146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:29.963389 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:29.963396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:29.965185 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:29.965200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:29.965727 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:29.965733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:29.965989 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:29.966717 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:29.966742 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:29.968643 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:29.971038 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:29.971046 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:29.971068 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:29.971416 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:29.971798 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:29.971804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:29.971809 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:29.972030 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:29.972639 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:29.972646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:29.973075 (+ 429us) client_negotiation.cc:769] Sending connection context
0926 07:19:29.973307 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 07:19:29.973660 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":124,"threads_started":1}
I20260926 07:19:29.976163 32357 heartbeater.cc:347] Connected to a master server at 127.28.235.254:46415
I20260926 07:19:29.976469 32357 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:29.977202 32357 heartbeater.cc:511] Master 127.28.235.254:46415 requested a full tablet report, sending...
I20260926 07:19:29.978502 32030 ts_manager.cc:194] Registered new tserver with Master: f34e50688cc446b1abfce800672cb09e (127.28.235.194:45601)
I20260926 07:19:29.979177 32030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:35233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:19:29.993331 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46415
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:30.109253 32362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:30.109578 32362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:30.109658 32362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:30.114112 32362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:30.114301 32362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:30.114460 32362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:30.120280 32362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:30.121491 32362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:30.122611 32362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:30.132797 32370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:30.132800 32368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:30.132797 32367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:30.133319 32362 server_base.cc:1034] running on GCE node
I20260926 07:19:30.133692 32362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:30.134366 32362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:30.135587 32362 hybrid_clock.cc:648] HybridClock initialized: now 1790407170135548 us; error 52 us; skew 500 ppm
Sep 26 07:19:30 dist-test-slave-7zlw krb5kdc[31987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407170, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:30.138725 32362 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:19:30.139756 32362 webserver.cc:460] Webserver started at http://127.28.235.195:33521/ using document root <none> and password file <none>
I20260926 07:19:30.140357 32362 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:30.140465 32362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:30.140714 32362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:30.142954 32362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "81e2b2e33f8b43ebb0948f89ec6cc374"
format_stamp: "Formatted at 2026-09-26 07:19:30 on dist-test-slave-7zlw"
server_key: "1945c36a4cb2d9227d867cc819c1b704"
server_key_iv: "18dd989f951d7b70525e10333751ad09"
server_key_version: "encryptionkey@0"
I20260926 07:19:30.143582 32362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "81e2b2e33f8b43ebb0948f89ec6cc374"
format_stamp: "Formatted at 2026-09-26 07:19:30 on dist-test-slave-7zlw"
server_key: "1945c36a4cb2d9227d867cc819c1b704"
server_key_iv: "18dd989f951d7b70525e10333751ad09"
server_key_version: "encryptionkey@0"
I20260926 07:19:30.147733 32362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:19:30.150512 32377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:30.151839 32362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:19:30.151984 32362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "81e2b2e33f8b43ebb0948f89ec6cc374"
format_stamp: "Formatted at 2026-09-26 07:19:30 on dist-test-slave-7zlw"
server_key: "1945c36a4cb2d9227d867cc819c1b704"
server_key_iv: "18dd989f951d7b70525e10333751ad09"
server_key_version: "encryptionkey@0"
I20260926 07:19:30.152107 32362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:30.175092 32362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:30.178897 32362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:30.179157 32362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:30.179803 32362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:30.180666 32362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:30.180737 32362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:30.180812 32362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:30.180855 32362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:30.190038 32362 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:37591
I20260926 07:19:30.190063 32490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:37591 every 8 connection(s)
I20260926 07:19:30.191077 32362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:30.200680 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 32362
I20260926 07:19:30.200850 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
Sep 26 07:19:30 dist-test-slave-7zlw krb5kdc[31987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407170, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:30.201259 29615 external_mini_cluster.cc:1428] Setting key 336fe9406698f30857ac56e233eb9d2e
I20260926 07:19:30.206416 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:30.193063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:45263 (local address 127.28.235.254:46415)
0926 07:19:30.193214 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:30.193218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:30.193978 (+ 760us) server_negotiation.cc:407] Connection header received
0926 07:19:30.195371 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:30.195374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:30.195434 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:30.195550 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:30.196765 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:30.197451 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:30.198636 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:30.198869 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:30.198982 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:30.201894 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:30.201898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:30.201900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:30.201926 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:30.203664 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:30.204223 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:30.204229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:30.204231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:30.204322 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:30.204711 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:30.204717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:30.204719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:30.204929 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:30.205068 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:30.205827 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 07:19:30.206153 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:19:30.207064 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:30.193375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:46415 (local address 127.28.235.195:45263)
0926 07:19:30.193826 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:30.193869 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:30.195161 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:30.195760 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:30.195767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:30.196134 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:30.196617 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:30.196630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:30.197603 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:30.197606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:30.198492 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:30.198501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:30.199034 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:30.199039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:30.199211 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:30.199808 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:30.199823 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:30.201699 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:30.203790 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:30.203797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:30.203813 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:30.204107 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:30.204432 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:30.204435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:30.204437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:30.204603 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:30.205084 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:30.205093 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:30.205435 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 07:19:30.205702 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 07:19:30.206080 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
I20260926 07:19:30.208367 32491 heartbeater.cc:347] Connected to a master server at 127.28.235.254:46415
I20260926 07:19:30.208643 32491 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:30.209214 32491 heartbeater.cc:511] Master 127.28.235.254:46415 requested a full tablet report, sending...
I20260926 07:19:30.210268 32030 ts_manager.cc:194] Registered new tserver with Master: 81e2b2e33f8b43ebb0948f89ec6cc374 (127.28.235.195:37591)
I20260926 07:19:30.210886 32030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:45263
I20260926 07:19:30.215181 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:19:30.229417 32071 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:19:30.227605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.28.235.254:46415)
0926 07:19:30.227745 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:30.227750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:30.228021 (+ 271us) server_negotiation.cc:407] Connection header received
0926 07:19:30.228224 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:30.228227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:30.228276 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:30.228362 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:30.229282 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51118: BlockingRecv error: recv got EOF from 127.0.0.1:51118 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:19:30.230523 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 32094
I20260926 07:19:30.236567 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 32228
I20260926 07:19:30.242400 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 32362
I20260926 07:19:30.248420 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 32000
2026-09-26T07:19:30Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2237 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:30 dist-test-slave-7zlw krb5kdc[32503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:30 dist-test-slave-7zlw krb5kdc[32503](info): set up 2 sockets
Sep 26 07:19:30 dist-test-slave-7zlw krb5kdc[32503](info): commencing operation
krb5kdc: starting...
W20260926 07:19:31.301127 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:31 dist-test-slave-7zlw krb5kdc[32503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:31.429414 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43707
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:32831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:43707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:31.541525 32516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:31.541879 32516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:31.541956 32516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:31.546159 32516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:31.546268 32516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:31.546303 32516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:31.546343 32516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:31.546375 32516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:31.551828 32516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:32831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:43707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:31.553033 32516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:31.554116 32516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:31.562955 32522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:31.563087 32521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:31.563172 32516 server_base.cc:1034] running on GCE node
W20260926 07:19:31.562976 32524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:31.563936 32516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:31.565186 32516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:31.566386 32516 hybrid_clock.cc:648] HybridClock initialized: now 1790407171566368 us; error 32 us; skew 500 ppm
Sep 26 07:19:31 dist-test-slave-7zlw krb5kdc[32503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407171, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:31.588250 32516 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:31.589358 32516 webserver.cc:460] Webserver started at http://127.28.235.254:43149/ using document root <none> and password file <none>
I20260926 07:19:31.590179 32516 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:31.590270 32516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:31.590536 32516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:31.592756 32516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "43d93eb1ede945fd84b76615aafe7835"
format_stamp: "Formatted at 2026-09-26 07:19:31 on dist-test-slave-7zlw"
server_key: "cbc6af78c052de07c784cb1575d5b47a"
server_key_iv: "573ed54bfb373aa66ec0be8f052a11f2"
server_key_version: "encryptionkey@0"
I20260926 07:19:31.593382 32516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "43d93eb1ede945fd84b76615aafe7835"
format_stamp: "Formatted at 2026-09-26 07:19:31 on dist-test-slave-7zlw"
server_key: "cbc6af78c052de07c784cb1575d5b47a"
server_key_iv: "573ed54bfb373aa66ec0be8f052a11f2"
server_key_version: "encryptionkey@0"
I20260926 07:19:31.597571 32516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:31.600260 32531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:31.601692 32516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:31.601858 32516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "43d93eb1ede945fd84b76615aafe7835"
format_stamp: "Formatted at 2026-09-26 07:19:31 on dist-test-slave-7zlw"
server_key: "cbc6af78c052de07c784cb1575d5b47a"
server_key_iv: "573ed54bfb373aa66ec0be8f052a11f2"
server_key_version: "encryptionkey@0"
I20260926 07:19:31.601985 32516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:31.633489 32516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:31.636914 32516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:31.637248 32516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:31.645171 32516 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:43707
I20260926 07:19:31.645557 32583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:43707 every 8 connection(s)
I20260926 07:19:31.646317 32516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:31.646876 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 32516
I20260926 07:19:31.647009 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:31.647279 29615 external_mini_cluster.cc:1428] Setting key e1ec8552ea78f42dedaee13f5fff9e50
I20260926 07:19:31.649936 32584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:31.657160 32584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835: Bootstrap starting.
Sep 26 07:19:31 dist-test-slave-7zlw krb5kdc[32503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:31.659858 32584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:31.660804 32584 log.cc:824] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:31.662925 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:31.648700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.28.235.254:43707)
0926 07:19:31.649183 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:31.649202 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:31.649245 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:19:31.650147 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:31.650173 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:31.650608 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:31.650982 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:31.651730 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:31.653310 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:31.654293 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:31.654994 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:31.655168 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:31.657927 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:31.657932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:31.657947 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:31.657976 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:31.659724 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:31.660224 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:31.660228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:31.660233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:31.660342 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:31.660668 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:31.660671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:31.660672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:31.661271 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:31.661489 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:31.661855 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 07:19:31.662172 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":123,"threads_started":1}
I20260926 07:19:31.663085 32584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835: No bootstrap required, opened a new log
I20260926 07:19:31.666115 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:19:31.667408 32584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d93eb1ede945fd84b76615aafe7835" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43707 } }
I20260926 07:19:31.667646 32584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:31.667749 32584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43d93eb1ede945fd84b76615aafe7835, State: Initialized, Role: FOLLOWER
I20260926 07:19:31.668200 32584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d93eb1ede945fd84b76615aafe7835" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43707 } }
I20260926 07:19:31.668357 32584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:31.668422 32584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:31.668521 32584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:31.669575 32584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d93eb1ede945fd84b76615aafe7835" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43707 } }
I20260926 07:19:31.669960 32584 leader_election.cc:302] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43d93eb1ede945fd84b76615aafe7835; no voters:
I20260926 07:19:31.670347 32584 leader_election.cc:288] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:31.670433 32590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:31.670667 32590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [term 1 LEADER]: Becoming Leader. State: Replica: 43d93eb1ede945fd84b76615aafe7835, State: Running, Role: LEADER
I20260926 07:19:31.671059 32590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d93eb1ede945fd84b76615aafe7835" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43707 } }
I20260926 07:19:31.671768 32584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:31.672866 32591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43d93eb1ede945fd84b76615aafe7835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d93eb1ede945fd84b76615aafe7835" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43707 } } }
I20260926 07:19:31.672995 32591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:31.673424 32592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43d93eb1ede945fd84b76615aafe7835. Latest consensus state: current_term: 1 leader_uuid: "43d93eb1ede945fd84b76615aafe7835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d93eb1ede945fd84b76615aafe7835" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43707 } } }
I20260926 07:19:31.673591 32592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:31.674337 32600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:31.677655 32600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:31.683807 32600 catalog_manager.cc:1347] Generated new cluster ID: 9570e1d9fde24c30917dd2da04eae8b9
I20260926 07:19:31.683880 32600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:31.686349 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:31.679463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.28.235.254:43707)
0926 07:19:31.679605 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:31.679609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:31.679979 (+ 370us) server_negotiation.cc:407] Connection header received
0926 07:19:31.680145 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:31.680148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:31.680202 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:31.680285 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:31.680928 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:31.682266 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:31.683544 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:31.683919 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:31.683999 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:31.685308 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:31.685312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:31.685315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:31.685870 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:31.685906 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:31.685928 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:31.686033 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:31.686111 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 07:19:31.686189 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:19:31.700627 32600 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:31.701656 32600 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:31.707293 32600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43d93eb1ede945fd84b76615aafe7835: Generated new TSK 0
I20260926 07:19:31.708014 32600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:19:31.718094 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 32516
2026-09-26T07:19:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1468 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:31 dist-test-slave-7zlw krb5kdc[32617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:31 dist-test-slave-7zlw krb5kdc[32617](info): set up 2 sockets
Sep 26 07:19:31 dist-test-slave-7zlw krb5kdc[32617](info): commencing operation
krb5kdc: starting...
W20260926 07:19:32.761195 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:32 dist-test-slave-7zlw krb5kdc[32617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:32.893297 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43391
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:43391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:33.009706 32630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:33.010035 32630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:33.010114 32630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:33.014490 32630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:33.014592 32630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:33.014631 32630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:33.014667 32630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:33.014700 32630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:33.020588 32630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:43391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:33.021869 32630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:33.023007 32630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:33.032248 32636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:33.032284 32635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:33.032282 32638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:33.032691 32630 server_base.cc:1034] running on GCE node
I20260926 07:19:33.033299 32630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:33.034435 32630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:33.035665 32630 hybrid_clock.cc:648] HybridClock initialized: now 1790407173035629 us; error 48 us; skew 500 ppm
Sep 26 07:19:33 dist-test-slave-7zlw krb5kdc[32617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:33.039026 32630 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:33.040026 32630 webserver.cc:460] Webserver started at http://127.28.235.254:44237/ using document root <none> and password file <none>
I20260926 07:19:33.040668 32630 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:33.040762 32630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:33.041070 32630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:33.043311 32630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "da2a3012f1934472b81e9e5b6554cf3d"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "c4d7ff3c4d7650f48b88396c6fd55c6d"
server_key_iv: "02d5a90c8c033aa57d02c4f6b8d0ace6"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.043993 32630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "da2a3012f1934472b81e9e5b6554cf3d"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "c4d7ff3c4d7650f48b88396c6fd55c6d"
server_key_iv: "02d5a90c8c033aa57d02c4f6b8d0ace6"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.048194 32630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:19:33.051009 32645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:33.052414 32630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:33.052610 32630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "da2a3012f1934472b81e9e5b6554cf3d"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "c4d7ff3c4d7650f48b88396c6fd55c6d"
server_key_iv: "02d5a90c8c033aa57d02c4f6b8d0ace6"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.052752 32630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:33.071762 32630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:33.075299 32630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:33.075569 32630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:33.082948 32630 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:43391
I20260926 07:19:33.082962 32697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:43391 every 8 connection(s)
I20260926 07:19:33.083938 32630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:33.087042 32698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:33.089908 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 32630
I20260926 07:19:33.090122 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:33.090405 29615 external_mini_cluster.cc:1428] Setting key eefdd516675c7adea1a2134645ff7647
I20260926 07:19:33.093631 32698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d: Bootstrap starting.
I20260926 07:19:33.096549 32698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:33.097456 32698 log.cc:824] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:33.099757 32698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d: No bootstrap required, opened a new log
Sep 26 07:19:33 dist-test-slave-7zlw krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:33.104480 32698 raft_consensus.cc:348] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2a3012f1934472b81e9e5b6554cf3d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43391 } }
I20260926 07:19:33.104816 32698 raft_consensus.cc:374] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:33.104876 32698 raft_consensus.cc:729] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da2a3012f1934472b81e9e5b6554cf3d, State: Initialized, Role: FOLLOWER
I20260926 07:19:33.105537 32698 consensus_queue.cc:260] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2a3012f1934472b81e9e5b6554cf3d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43391 } }
I20260926 07:19:33.105715 32698 raft_consensus.cc:388] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:33.105777 32698 raft_consensus.cc:482] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:33.105885 32698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:33.107180 32698 raft_consensus.cc:504] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2a3012f1934472b81e9e5b6554cf3d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43391 } }
I20260926 07:19:33.107571 32698 leader_election.cc:302] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da2a3012f1934472b81e9e5b6554cf3d; no voters:
I20260926 07:19:33.107867 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:33.091924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.28.235.254:43391)
0926 07:19:33.092411 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:33.092434 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:33.092495 (+ 61us) server_negotiation.cc:407] Connection header received
0926 07:19:33.093365 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:33.093389 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:33.093828 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:33.094210 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:33.094938 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.096455 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.097538 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.098179 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.098395 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:33.101291 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:33.101299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:33.101317 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:33.101354 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:33.104245 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:33.104756 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:33.104765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:33.104774 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:33.104934 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:33.105346 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:33.105350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:33.105352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:33.105918 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:33.106152 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:33.106416 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 07:19:33.106752 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":137,"threads_started":1}
I20260926 07:19:33.107957 32698 leader_election.cc:288] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:33.108227 32703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:33.108501 32703 raft_consensus.cc:686] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [term 1 LEADER]: Becoming Leader. State: Replica: da2a3012f1934472b81e9e5b6554cf3d, State: Running, Role: LEADER
I20260926 07:19:33.108839 32703 consensus_queue.cc:237] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2a3012f1934472b81e9e5b6554cf3d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43391 } }
I20260926 07:19:33.110637 32698 sys_catalog.cc:564] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:33.110586 32704 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da2a3012f1934472b81e9e5b6554cf3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2a3012f1934472b81e9e5b6554cf3d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43391 } } }
I20260926 07:19:33.110780 32704 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:33.110786 32705 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader da2a3012f1934472b81e9e5b6554cf3d. Latest consensus state: current_term: 1 leader_uuid: "da2a3012f1934472b81e9e5b6554cf3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2a3012f1934472b81e9e5b6554cf3d" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43391 } } }
I20260926 07:19:33.110893 32705 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:33.111217 32707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:33.114583 32707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:33.121079 32707 catalog_manager.cc:1347] Generated new cluster ID: d1964221994b4bca8c115207324237a5
I20260926 07:19:33.121170 32707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:33.130538 32707 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:33.131484 32707 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:33.137001 32707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da2a3012f1934472b81e9e5b6554cf3d: Generated new TSK 0
I20260926 07:19:33.137777 32707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:19:33.172279 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43391
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:33.288429 32724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:33.288707 32724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:33.288751 32724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:33.293372 32724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:33.293455 32724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:33.293529 32724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:33.299804 32724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:33.300905 32724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:33.302006 32724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:33.312299 32730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:33.312367 32729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:33.312467 32732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:33.312350 32724 server_base.cc:1034] running on GCE node
I20260926 07:19:33.312986 32724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:33.313814 32724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:33.314992 32724 hybrid_clock.cc:648] HybridClock initialized: now 1790407173314975 us; error 27 us; skew 500 ppm
Sep 26 07:19:33 dist-test-slave-7zlw krb5kdc[32617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:33.318342 32724 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:19:33.319353 32724 webserver.cc:460] Webserver started at http://127.28.235.193:40935/ using document root <none> and password file <none>
I20260926 07:19:33.320017 32724 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:33.320120 32724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:33.320376 32724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:33.322736 32724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "a1fcfb93c054436ea000245d59322a16"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "a763f01e69b7ba7ec1b1f6cee2bff021"
server_key_iv: "69ae8807dd320c517775a33bfa1a8c5d"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.323377 32724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "a1fcfb93c054436ea000245d59322a16"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "a763f01e69b7ba7ec1b1f6cee2bff021"
server_key_iv: "69ae8807dd320c517775a33bfa1a8c5d"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.327767 32724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:19:33.330597 32739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:33.331990 32724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:19:33.332150 32724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "a1fcfb93c054436ea000245d59322a16"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "a763f01e69b7ba7ec1b1f6cee2bff021"
server_key_iv: "69ae8807dd320c517775a33bfa1a8c5d"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.332279 32724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:33.353255 32724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:33.356765 32724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:33.356999 32724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:33.357761 32724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:33.358676 32724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:33.358755 32724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:33.358834 32724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:33.358881 32724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:33.369172 32724 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:36223
I20260926 07:19:33.369215 384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:36223 every 8 connection(s)
I20260926 07:19:33.370160 32724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:33.380272 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 32724
I20260926 07:19:33.380409 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
Sep 26 07:19:33 dist-test-slave-7zlw krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:33.380793 29615 external_mini_cluster.cc:1428] Setting key 8d49da34439d9054eb9bdce4c895da0b
I20260926 07:19:33.385568 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:33.372057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:56785 (local address 127.28.235.254:43391)
0926 07:19:33.372264 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:33.372268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:33.373292 (+ 1024us) server_negotiation.cc:407] Connection header received
0926 07:19:33.374823 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:33.374827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:33.374891 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:33.374977 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:33.376193 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.376795 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.378101 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.378370 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.378493 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:33.381481 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:33.381484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:33.381486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:33.381514 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:33.383191 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:33.383836 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:33.383839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:33.383840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:33.383918 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:33.384329 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:33.384333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:33.384335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:33.384528 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:33.384641 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:33.385251 (+ 610us) server_negotiation.cc:299] Negotiation successful
0926 07:19:33.385384 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:19:33.386449 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:33.372522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43391 (local address 127.28.235.193:56785)
0926 07:19:33.372981 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:33.373047 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:33.374591 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:33.375160 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:33.375167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:33.375537 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:33.376030 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:33.376044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.376951 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.376954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:33.377955 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:33.377964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.378532 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.378537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:33.378726 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:33.379381 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:33.379399 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:33.381303 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:33.383321 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:33.383330 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:33.383349 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:33.383726 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:33.384046 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:33.384049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:33.384052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:33.384217 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:33.384640 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:33.384649 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:33.384931 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 07:19:33.385194 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 07:19:33.385550 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260926 07:19:33.387856 385 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43391
I20260926 07:19:33.388134 385 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:33.388665 385 heartbeater.cc:511] Master 127.28.235.254:43391 requested a full tablet report, sending...
I20260926 07:19:33.390522 32662 ts_manager.cc:194] Registered new tserver with Master: a1fcfb93c054436ea000245d59322a16 (127.28.235.193:36223)
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:19:33.391788 32662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:56785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:19:33.413252 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43391
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:33.527393 390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:33.527664 390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:33.527709 390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:33.531883 390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:33.531947 390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:33.532045 390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:33.537703 390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:33.538829 390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:33.539858 390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:33.549631 395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:33.549680 390 server_base.cc:1034] running on GCE node
W20260926 07:19:33.549631 396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:33.549659 398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:33.550343 390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:33.551064 390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:33.552289 390 hybrid_clock.cc:648] HybridClock initialized: now 1790407173552249 us; error 49 us; skew 500 ppm
Sep 26 07:19:33 dist-test-slave-7zlw krb5kdc[32617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:33.555258 390 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:19:33.556142 390 webserver.cc:460] Webserver started at http://127.28.235.194:44383/ using document root <none> and password file <none>
I20260926 07:19:33.556748 390 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:33.556859 390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:33.557140 390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:33.559247 390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "15907fd5d3d143679ae9e66dc7c732ef"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "4ef7c38ea8bb96030bcf912cd07673cb"
server_key_iv: "244d460241f3f6f3c7fe3fc5d7087e22"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.559829 390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "15907fd5d3d143679ae9e66dc7c732ef"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "4ef7c38ea8bb96030bcf912cd07673cb"
server_key_iv: "244d460241f3f6f3c7fe3fc5d7087e22"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.563784 390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:19:33.566368 405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:33.567767 390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:19:33.567922 390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "15907fd5d3d143679ae9e66dc7c732ef"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "4ef7c38ea8bb96030bcf912cd07673cb"
server_key_iv: "244d460241f3f6f3c7fe3fc5d7087e22"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.568053 390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:33.589812 390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:33.593124 390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:33.593320 390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:33.593916 390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:33.594749 390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:33.594820 390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:33.594894 390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:33.594937 390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:33.604182 390 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:37855
I20260926 07:19:33.604200 518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:37855 every 8 connection(s)
I20260926 07:19:33.605173 390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:33.610162 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 390
I20260926 07:19:33.610283 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:33.610579 29615 external_mini_cluster.cc:1428] Setting key 64dde9a48291bc2921e5bb06fa5c59e1
Sep 26 07:19:33 dist-test-slave-7zlw krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:19:33.621006 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:33.607247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:39765 (local address 127.28.235.254:43391)
0926 07:19:33.607406 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:33.607410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:33.608301 (+ 891us) server_negotiation.cc:407] Connection header received
0926 07:19:33.609732 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:33.609735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:33.609782 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:33.609890 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:33.611561 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.612308 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.613726 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.613974 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.614102 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:33.617163 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:33.617171 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:33.617174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:33.617198 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:33.618711 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:33.619259 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:33.619263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:33.619264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:33.619343 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:33.619720 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:33.619724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:33.619725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:33.619970 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:33.620146 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:33.620685 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 07:19:33.620819 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:19:33.621927 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:33.607611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43391 (local address 127.28.235.194:39765)
0926 07:19:33.608140 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:33.608185 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:33.609520 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:33.610017 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:33.610024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:33.610685 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:33.611353 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:33.611397 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.612436 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.612439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:33.613594 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:33.613604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.614139 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.614144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:33.614321 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:33.614989 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:33.615007 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:33.616909 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:33.618836 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:33.618842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:33.618858 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:33.619138 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:33.619446 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:33.619449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:33.619451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:33.619611 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:33.620150 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:33.620160 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:33.620421 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 07:19:33.620624 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 07:19:33.620983 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
I20260926 07:19:33.623230 519 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43391
I20260926 07:19:33.623489 519 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:33.623996 519 heartbeater.cc:511] Master 127.28.235.254:43391 requested a full tablet report, sending...
I20260926 07:19:33.625255 32662 ts_manager.cc:194] Registered new tserver with Master: 15907fd5d3d143679ae9e66dc7c732ef (127.28.235.194:37855)
I20260926 07:19:33.625896 32662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:39765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:19:33.641070 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43391
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:33.823995 524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:33.824260 524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:33.824303 524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:33.828774 524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:33.828845 524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:33.828915 524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:33.834465 524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:33.835541 524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:33.836627 524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:33.847229 529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:33.847229 532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:33.847427 524 server_base.cc:1034] running on GCE node
W20260926 07:19:33.847146 530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:33.848523 524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:33.849362 524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:33.850564 524 hybrid_clock.cc:648] HybridClock initialized: now 1790407173850525 us; error 43 us; skew 500 ppm
Sep 26 07:19:33 dist-test-slave-7zlw krb5kdc[32617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:33.853763 524 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:19:33.854733 524 webserver.cc:460] Webserver started at http://127.28.235.195:43893/ using document root <none> and password file <none>
I20260926 07:19:33.855361 524 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:33.855460 524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:33.855701 524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:33.858191 524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "11ca589c7ce941fdb2640974677e5d50"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "4208ba2538fcdc03048ed71598459478"
server_key_iv: "bd1123b3fba8aa658268a2ec9d8be4de"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.858778 524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "11ca589c7ce941fdb2640974677e5d50"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "4208ba2538fcdc03048ed71598459478"
server_key_iv: "bd1123b3fba8aa658268a2ec9d8be4de"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.862665 524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:19:33.865506 539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:33.866902 524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:33.867105 524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "11ca589c7ce941fdb2640974677e5d50"
format_stamp: "Formatted at 2026-09-26 07:19:33 on dist-test-slave-7zlw"
server_key: "4208ba2538fcdc03048ed71598459478"
server_key_iv: "bd1123b3fba8aa658268a2ec9d8be4de"
server_key_version: "encryptionkey@0"
I20260926 07:19:33.867236 524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:33.885356 524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:33.888896 524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:33.889137 524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:33.889770 524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:33.890667 524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:33.890736 524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:33.890808 524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:33.890852 524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:33.899986 524 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:39111
I20260926 07:19:33.900027 652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:39111 every 8 connection(s)
I20260926 07:19:33.900949 524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:33.909227 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 524
I20260926 07:19:33.909364 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:33.909657 29615 external_mini_cluster.cc:1428] Setting key 6822900f12d6f6292ea4fd3fb26fbe52
Sep 26 07:19:33 dist-test-slave-7zlw krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407173, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:33.916079 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:33.902941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:59885 (local address 127.28.235.254:43391)
0926 07:19:33.903092 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:33.903096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:33.903963 (+ 867us) server_negotiation.cc:407] Connection header received
0926 07:19:33.905488 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:33.905492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:33.905566 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:33.905643 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:33.906948 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.907492 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.908640 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.908853 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.908983 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:33.912265 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:33.912269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:33.912272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:33.912306 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:33.913828 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:33.914417 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:33.914422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:33.914424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:33.914509 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:33.914872 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:33.914876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:33.914878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:33.915070 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:33.915202 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:33.915783 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 07:19:33.915913 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:19:33.917130 655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:33.903322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43391 (local address 127.28.235.195:59885)
0926 07:19:33.903794 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:33.903838 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:33.905229 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:33.905880 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:33.905887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:33.906314 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:33.906809 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:33.906823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.907606 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.907609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:33.908515 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:33.908524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.909063 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.909068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:33.909244 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:33.909998 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:33.910027 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:33.912005 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:33.913955 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:33.913962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:33.913981 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:33.914297 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:33.914602 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:33.914616 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:33.914618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:33.914775 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:33.915202 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:33.915211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:33.915541 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 07:19:33.915745 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 07:19:33.916106 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
I20260926 07:19:33.918449 653 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43391
I20260926 07:19:33.918766 653 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:33.919308 653 heartbeater.cc:511] Master 127.28.235.254:43391 requested a full tablet report, sending...
I20260926 07:19:33.920542 32662 ts_manager.cc:194] Registered new tserver with Master: 11ca589c7ce941fdb2640974677e5d50 (127.28.235.195:39111)
I20260926 07:19:33.921274 32662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:59885
I20260926 07:19:33.924289 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:19:33 dist-test-slave-7zlw krb5kdc[32617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407173, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 07:19:33 dist-test-slave-7zlw krb5kdc[32617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407173, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:33.989929 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:33.980104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.28.235.254:43391)
0926 07:19:33.980253 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:33.980261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:33.980450 (+ 189us) server_negotiation.cc:407] Connection header received
0926 07:19:33.980639 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:33.980642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:33.980690 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:33.980782 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:33.981636 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.982182 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.983051 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:33.983274 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:33.983386 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:33.986024 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:33.986028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:33.986030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:33.986077 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:33.987944 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:33.988468 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:33.988495 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:33.988496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:33.988582 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:33.988923 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:33.988929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:33.988933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:33.989227 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:33.989377 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:33.989623 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 07:19:33.989758 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 07:19:33.990379 32662 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48848
I20260926 07:19:33.991484 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 32724
I20260926 07:19:33.998119 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 390
I20260926 07:19:34.003885 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 524
I20260926 07:19:34.009235 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 32630
2026-09-26T07:19:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2291 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:35 dist-test-slave-7zlw krb5kdc[665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:35 dist-test-slave-7zlw krb5kdc[665](info): set up 2 sockets
Sep 26 07:19:35 dist-test-slave-7zlw krb5kdc[665](info): commencing operation
krb5kdc: starting...
W20260926 07:19:36.788477 29615 mini_kdc.cc:121] Time spent starting KDC: real 2.757s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 07:19:36.949569 29615 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.161s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 07:19:37.215586 29615 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.266s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 07:19:37.445443 29615 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.230s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/test-user.keytab.
W20260926 07:19:37.718873 29615 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.273s user 0.001s sys 0.001s
Sep 26 07:19:37 dist-test-slave-7zlw krb5kdc[665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:37.769258 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:46089
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:46089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:37.886595 678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:37.886866 678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:37.886906 678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:37.890897 678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:37.890956 678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:37.890976 678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:37.890993 678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:37.891008 678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:37.896425 678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:46089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:46089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:37.897573 678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:37.898582 678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:37.907810 683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:37.907859 686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:37.907832 684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:37.908082 678 server_base.cc:1034] running on GCE node
I20260926 07:19:37.908674 678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:37.909909 678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:37.911067 678 hybrid_clock.cc:648] HybridClock initialized: now 1790407177911036 us; error 44 us; skew 500 ppm
Sep 26 07:19:37 dist-test-slave-7zlw krb5kdc[665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407177, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:37.920207 678 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:37.921270 678 webserver.cc:460] Webserver started at http://127.28.235.254:38445/ using document root <none> and password file <none>
I20260926 07:19:37.921837 678 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:37.921896 678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:37.922070 678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:37.924219 678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "da9f63daf9d94a419f884390c3ef1123"
format_stamp: "Formatted at 2026-09-26 07:19:37 on dist-test-slave-7zlw"
server_key: "b28686f55972f675d93be0f34f207cb1"
server_key_iv: "bb23c34d8eebf0f68885e4ad27455471"
server_key_version: "encryptionkey@0"
I20260926 07:19:37.924780 678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "da9f63daf9d94a419f884390c3ef1123"
format_stamp: "Formatted at 2026-09-26 07:19:37 on dist-test-slave-7zlw"
server_key: "b28686f55972f675d93be0f34f207cb1"
server_key_iv: "bb23c34d8eebf0f68885e4ad27455471"
server_key_version: "encryptionkey@0"
I20260926 07:19:37.929129 678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:19:37.931842 693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:37.933254 678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:37.933390 678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "da9f63daf9d94a419f884390c3ef1123"
format_stamp: "Formatted at 2026-09-26 07:19:37 on dist-test-slave-7zlw"
server_key: "b28686f55972f675d93be0f34f207cb1"
server_key_iv: "bb23c34d8eebf0f68885e4ad27455471"
server_key_version: "encryptionkey@0"
I20260926 07:19:37.933499 678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:37.954797 678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:37.958205 678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:37.958411 678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:37.966050 678 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:46089
I20260926 07:19:37.966059 745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:46089 every 8 connection(s)
I20260926 07:19:37.967104 678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:37.970113 746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:37.975668 746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123: Bootstrap starting.
I20260926 07:19:37.976699 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 678
I20260926 07:19:37.976841 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:37.977175 29615 external_mini_cluster.cc:1428] Setting key 98acacdf7358dc5ff311cad9650a569b
I20260926 07:19:37.978749 746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:37.979480 746 log.cc:824] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:37.981967 746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123: No bootstrap required, opened a new log
Sep 26 07:19:37 dist-test-slave-7zlw krb5kdc[665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:37.986984 746 raft_consensus.cc:348] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f63daf9d94a419f884390c3ef1123" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46089 } }
I20260926 07:19:37.987243 746 raft_consensus.cc:374] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:37.987342 746 raft_consensus.cc:729] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9f63daf9d94a419f884390c3ef1123, State: Initialized, Role: FOLLOWER
I20260926 07:19:37.987886 746 consensus_queue.cc:260] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f63daf9d94a419f884390c3ef1123" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46089 } }
I20260926 07:19:37.988080 746 raft_consensus.cc:388] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:37.988189 746 raft_consensus.cc:482] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:37.988305 746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:37.989398 746 raft_consensus.cc:504] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f63daf9d94a419f884390c3ef1123" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46089 } }
I20260926 07:19:37.989789 746 leader_election.cc:302] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da9f63daf9d94a419f884390c3ef1123; no voters:
I20260926 07:19:37.990161 746 leader_election.cc:288] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:37.990266 751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:37.990463 751 raft_consensus.cc:686] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [term 1 LEADER]: Becoming Leader. State: Replica: da9f63daf9d94a419f884390c3ef1123, State: Running, Role: LEADER
I20260926 07:19:37.990830 751 consensus_queue.cc:237] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f63daf9d94a419f884390c3ef1123" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46089 } }
I20260926 07:19:37.991551 746 sys_catalog.cc:564] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:37.992599 752 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da9f63daf9d94a419f884390c3ef1123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f63daf9d94a419f884390c3ef1123" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46089 } } }
I20260926 07:19:37.992754 752 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:37.993064 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:37.978364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.28.235.254:46089)
0926 07:19:37.978797 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:37.978825 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:37.978872 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:19:37.979512 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:37.979529 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:37.980001 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:37.980369 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:37.981445 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:37.982525 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:37.983725 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:37.984233 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:37.984349 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:37.987367 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:37.987372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:37.987386 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:37.987415 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:37.989474 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:37.989946 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:37.989952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:37.989958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:37.990081 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:37.990409 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:37.990413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:37.990415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:37.990963 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:37.991157 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:37.991692 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 07:19:37.992048 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":136,"threads_started":1}
I20260926 07:19:37.993124 753 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da9f63daf9d94a419f884390c3ef1123. Latest consensus state: current_term: 1 leader_uuid: "da9f63daf9d94a419f884390c3ef1123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f63daf9d94a419f884390c3ef1123" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46089 } } }
I20260926 07:19:37.993649 753 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123 [sys.catalog]: This master's current role is: LEADER
W20260926 07:19:37.997164 767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:19:37.997246 767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:19:37.997326 760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:37.998224 760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:38.008497 760 catalog_manager.cc:1347] Generated new cluster ID: 1a115f69232343cc8ee7bd2e5c46230d
I20260926 07:19:38.008601 760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:38.036676 760 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:38.037838 760 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:38.054803 760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da9f63daf9d94a419f884390c3ef1123: Generated new TSK 0
I20260926 07:19:38.055598 760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:19:38.096439 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46089
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:38.208639 772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:38.208912 772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:38.208957 772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:38.213153 772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:38.213214 772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:38.213275 772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:38.218530 772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:38.219625 772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:38.220607 772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:38.230643 778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:38.230643 780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:38.230656 777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:38.230739 772 server_base.cc:1034] running on GCE node
I20260926 07:19:38.231326 772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:38.231979 772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:38.233186 772 hybrid_clock.cc:648] HybridClock initialized: now 1790407178233167 us; error 27 us; skew 500 ppm
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:38.235985 772 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:19:38.236915 772 webserver.cc:460] Webserver started at http://127.28.235.193:37853/ using document root <none> and password file <none>
I20260926 07:19:38.237599 772 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:38.237705 772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:38.237996 772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:38.240163 772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "90d7645c09704f75bcd6b2dcb4f1034f"
format_stamp: "Formatted at 2026-09-26 07:19:38 on dist-test-slave-7zlw"
server_key: "67205c9d470a752358426b7066e3fd27"
server_key_iv: "63d4e53a03fa442b5ac0691a84e4bf52"
server_key_version: "encryptionkey@0"
I20260926 07:19:38.240769 772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "90d7645c09704f75bcd6b2dcb4f1034f"
format_stamp: "Formatted at 2026-09-26 07:19:38 on dist-test-slave-7zlw"
server_key: "67205c9d470a752358426b7066e3fd27"
server_key_iv: "63d4e53a03fa442b5ac0691a84e4bf52"
server_key_version: "encryptionkey@0"
I20260926 07:19:38.244895 772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:19:38.247536 787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:38.248736 772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:19:38.248875 772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "90d7645c09704f75bcd6b2dcb4f1034f"
format_stamp: "Formatted at 2026-09-26 07:19:38 on dist-test-slave-7zlw"
server_key: "67205c9d470a752358426b7066e3fd27"
server_key_iv: "63d4e53a03fa442b5ac0691a84e4bf52"
server_key_version: "encryptionkey@0"
I20260926 07:19:38.249001 772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:38.261866 772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:38.265120 772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:38.265337 772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:38.265987 772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:38.266873 772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:38.266954 772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:38.267045 772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:38.267100 772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:38.276196 772 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:42729
I20260926 07:19:38.276254 900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:42729 every 8 connection(s)
I20260926 07:19:38.277230 772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:38.283229 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 772
I20260926 07:19:38.283366 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:38.283702 29615 external_mini_cluster.cc:1428] Setting key 4d0a76b76d205f097268415a4cc9d70d
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:38.293236 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:38.279110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:34321 (local address 127.28.235.254:46089)
0926 07:19:38.279287 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:38.279292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:38.280249 (+ 957us) server_negotiation.cc:407] Connection header received
0926 07:19:38.281748 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:38.281751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:38.281805 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:38.281884 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:38.283166 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.283959 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.285570 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.285957 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.286102 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:38.289198 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:38.289203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:38.289206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:38.289240 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:38.290924 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:38.291522 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:38.291530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:38.291531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:38.291610 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:38.291969 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:38.291972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:38.291974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:38.292234 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:38.292359 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:38.292903 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 07:19:38.293065 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:19:38.294304 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:38.279577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:46089 (local address 127.28.235.193:34321)
0926 07:19:38.280074 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:38.280118 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:38.281523 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:38.282063 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:38.282071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:38.282468 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:38.282997 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:38.283012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.284105 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.284109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:38.285426 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:38.285437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.286117 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.286120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:38.286279 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:38.286935 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:38.286965 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:38.288954 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:38.291053 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:38.291060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:38.291078 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:38.291411 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:38.291712 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:38.291716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:38.291718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:38.291871 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:38.292357 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:38.292365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:38.292656 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 07:19:38.292848 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 07:19:38.293300 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
I20260926 07:19:38.295766 901 heartbeater.cc:347] Connected to a master server at 127.28.235.254:46089
I20260926 07:19:38.296172 901 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:19:38.296828 901 heartbeater.cc:511] Master 127.28.235.254:46089 requested a full tablet report, sending...
I20260926 07:19:38.298627 710 ts_manager.cc:194] Registered new tserver with Master: 90d7645c09704f75bcd6b2dcb4f1034f (127.28.235.193:42729)
I20260926 07:19:38.299896 710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:34321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:19:38.312894 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46089
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:38.425096 906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:38.425419 906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:38.425498 906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:38.430146 906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:38.430238 906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:38.430326 906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:38.435832 906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:38.437013 906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:38.438285 906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:38.448127 911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:38.448184 912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:38.448184 914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:38.448365 906 server_base.cc:1034] running on GCE node
I20260926 07:19:38.448827 906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:38.449555 906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:38.450738 906 hybrid_clock.cc:648] HybridClock initialized: now 1790407178450725 us; error 40 us; skew 500 ppm
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:38.453778 906 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:19:38.454622 906 webserver.cc:460] Webserver started at http://127.28.235.194:34735/ using document root <none> and password file <none>
I20260926 07:19:38.455212 906 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:38.455300 906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:38.455528 906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:38.457620 906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "5b812306c35b4c0fa6a2d29b3043cd74"
format_stamp: "Formatted at 2026-09-26 07:19:38 on dist-test-slave-7zlw"
server_key: "fefa2cba0f92e1111da974b738b32cfc"
server_key_iv: "3b298ca87dca81dc5af9299505d0cb21"
server_key_version: "encryptionkey@0"
I20260926 07:19:38.458176 906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "5b812306c35b4c0fa6a2d29b3043cd74"
format_stamp: "Formatted at 2026-09-26 07:19:38 on dist-test-slave-7zlw"
server_key: "fefa2cba0f92e1111da974b738b32cfc"
server_key_iv: "3b298ca87dca81dc5af9299505d0cb21"
server_key_version: "encryptionkey@0"
I20260926 07:19:38.461952 906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:38.464376 921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:38.465749 906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:19:38.465881 906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "5b812306c35b4c0fa6a2d29b3043cd74"
format_stamp: "Formatted at 2026-09-26 07:19:38 on dist-test-slave-7zlw"
server_key: "fefa2cba0f92e1111da974b738b32cfc"
server_key_iv: "3b298ca87dca81dc5af9299505d0cb21"
server_key_version: "encryptionkey@0"
I20260926 07:19:38.465991 906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:38.490032 906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:38.493364 906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:38.493603 906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:38.494217 906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:38.495014 906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:38.495078 906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:38.495152 906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:38.495198 906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:38.504231 906 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:43859
I20260926 07:19:38.504254 1034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:43859 every 8 connection(s)
I20260926 07:19:38.505204 906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:38.510003 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 906
I20260926 07:19:38.510133 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:38.510455 29615 external_mini_cluster.cc:1428] Setting key d4d0069025b8cb3b37835e9d129906d6
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:19:38.521283 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:38.507161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:49703 (local address 127.28.235.254:46089)
0926 07:19:38.507336 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:38.507341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:38.508230 (+ 889us) server_negotiation.cc:407] Connection header received
0926 07:19:38.509647 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:38.509650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:38.509708 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:38.509782 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:38.511519 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.512237 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.514076 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.514474 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.514595 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:38.517179 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:38.517187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:38.517192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:38.517227 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:38.518984 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:38.519511 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:38.519516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:38.519520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:38.519614 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:38.519957 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:38.519961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:38.519963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:38.520239 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:38.520349 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:38.520998 (+ 649us) server_negotiation.cc:299] Negotiation successful
0926 07:19:38.521154 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":70}
I20260926 07:19:38.522239 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:38.507589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:46089 (local address 127.28.235.194:49703)
0926 07:19:38.508067 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:38.508110 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:38.509451 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:38.510002 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:38.510012 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:38.510534 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:38.511372 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:38.511390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.512374 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.512378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:38.513933 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:38.513949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.514625 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.514628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:38.514776 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:38.515427 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:38.515444 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:38.516943 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:38.519111 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:38.519115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:38.519128 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:38.519392 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:38.519715 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:38.519717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:38.519718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:38.519847 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:38.520374 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:38.520380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:38.520704 (+ 324us) client_negotiation.cc:769] Sending connection context
0926 07:19:38.520896 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 07:19:38.521262 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
I20260926 07:19:38.523653 1035 heartbeater.cc:347] Connected to a master server at 127.28.235.254:46089
I20260926 07:19:38.523881 1035 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:38.524389 1035 heartbeater.cc:511] Master 127.28.235.254:46089 requested a full tablet report, sending...
I20260926 07:19:38.525558 710 ts_manager.cc:194] Registered new tserver with Master: 5b812306c35b4c0fa6a2d29b3043cd74 (127.28.235.194:43859)
I20260926 07:19:38.526145 710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:49703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:19:38.540560 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46089
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:38.652495 1040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:38.652797 1040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:38.652879 1040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:38.657063 1040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:38.657152 1040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:38.657251 1040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:38.662555 1040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:46089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:38.663697 1040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:38.664847 1040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:38.675005 1046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:38.675009 1048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:38.675041 1045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:38.675247 1040 server_base.cc:1034] running on GCE node
I20260926 07:19:38.675771 1040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:38.676424 1040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:38.677629 1040 hybrid_clock.cc:648] HybridClock initialized: now 1790407178677593 us; error 42 us; skew 500 ppm
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:38.680449 1040 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:19:38.681388 1040 webserver.cc:460] Webserver started at http://127.28.235.195:39389/ using document root <none> and password file <none>
I20260926 07:19:38.681967 1040 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:38.682112 1040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:38.682353 1040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:38.684394 1040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "31d25a02d0f14ed9b469b90dde205f01"
format_stamp: "Formatted at 2026-09-26 07:19:38 on dist-test-slave-7zlw"
server_key: "89c324ed8a187cd5c6ef1d6369765551"
server_key_iv: "c3675337f7062c3d08ec3c2ebb96e7da"
server_key_version: "encryptionkey@0"
I20260926 07:19:38.684984 1040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "31d25a02d0f14ed9b469b90dde205f01"
format_stamp: "Formatted at 2026-09-26 07:19:38 on dist-test-slave-7zlw"
server_key: "89c324ed8a187cd5c6ef1d6369765551"
server_key_iv: "c3675337f7062c3d08ec3c2ebb96e7da"
server_key_version: "encryptionkey@0"
I20260926 07:19:38.688723 1040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:38.691501 1055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:38.692770 1040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:38.692958 1040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "31d25a02d0f14ed9b469b90dde205f01"
format_stamp: "Formatted at 2026-09-26 07:19:38 on dist-test-slave-7zlw"
server_key: "89c324ed8a187cd5c6ef1d6369765551"
server_key_iv: "c3675337f7062c3d08ec3c2ebb96e7da"
server_key_version: "encryptionkey@0"
I20260926 07:19:38.693101 1040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:38.713824 1040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:38.717064 1040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:38.717283 1040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:38.717887 1040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:38.718742 1040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:38.718811 1040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:38.718878 1040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:38.718922 1040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:38.728523 1040 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:44591
I20260926 07:19:38.728549 1168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:44591 every 8 connection(s)
I20260926 07:19:38.729502 1040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:38.737392 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 1040
I20260926 07:19:38.737553 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:38.737941 29615 external_mini_cluster.cc:1428] Setting key a3e90ec7a03256ffecc53749435c7f7b
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407178, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:38.743657 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:38.731472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:39411 (local address 127.28.235.254:46089)
0926 07:19:38.731627 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:38.731631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:38.732498 (+ 867us) server_negotiation.cc:407] Connection header received
0926 07:19:38.733943 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:38.733947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:38.734006 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:38.734085 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:38.735277 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.735819 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.736991 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.737279 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.737401 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:38.740121 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:38.740124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:38.740127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:38.740167 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:38.741628 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:38.742157 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:38.742161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:38.742163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:38.742250 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:38.742590 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:38.742594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:38.742596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:38.742787 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:38.742891 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:38.743379 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 07:19:38.743499 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:19:38.744504 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:38.731883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:46089 (local address 127.28.235.195:39411)
0926 07:19:38.732345 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:38.732388 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:38.733729 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:38.734264 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:38.734272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:38.734632 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:38.735116 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:38.735130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.735951 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.735954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:38.736862 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:38.736872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.737440 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.737444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:38.737689 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:38.738409 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:38.738434 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:38.739944 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:38.741742 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:38.741748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:38.741764 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:38.742044 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:38.742341 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:38.742345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:38.742347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:38.742493 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:38.742889 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:38.742896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:38.743147 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 07:19:38.743340 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 07:19:38.743687 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260926 07:19:38.745715 1169 heartbeater.cc:347] Connected to a master server at 127.28.235.254:46089
I20260926 07:19:38.745980 1169 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:38.746528 1169 heartbeater.cc:511] Master 127.28.235.254:46089 requested a full tablet report, sending...
I20260926 07:19:38.747675 710 ts_manager.cc:194] Registered new tserver with Master: 31d25a02d0f14ed9b469b90dde205f01 (127.28.235.195:44591)
I20260926 07:19:38.748291 710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:39411
I20260926 07:19:38.752236 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:19:38.780556 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:38.769626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.28.235.193:42729)
0926 07:19:38.770024 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:38.770042 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:38.770069 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:19:38.770207 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:38.770217 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:38.770466 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:38.770613 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:38.771449 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.772366 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.773391 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.773821 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.773923 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:38.776773 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:38.776784 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:38.776801 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:38.776828 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:38.778550 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:38.779074 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:38.779080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:38.779081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:38.779178 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:38.779436 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:38.779438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:38.779439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:38.779729 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:38.779926 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:38.780076 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 07:19:38.780324 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":140,"threads_started":1}
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:38.792135 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:38.783665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.28.235.254:46089)
0926 07:19:38.783816 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:38.783820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:38.783834 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:19:38.783954 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:38.783957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:38.783999 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:38.784067 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:38.784658 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.785272 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.786099 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:38.786325 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:38.786416 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:38.788967 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:38.788970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:38.788972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:38.788997 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:38.790368 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:38.790763 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:38.790766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:38.790767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:38.790847 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:38.791149 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:38.791151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:38.791152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:38.791470 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:38.791581 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:38.791816 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 07:19:38.791918 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 07:19:38.792665 710 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53530
I20260926 07:19:38.793417 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 772
I20260926 07:19:38.799427 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 906
I20260926 07:19:38.804885 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 1040
I20260926 07:19:38.810456 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 678
2026-09-26T07:19:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4801 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[1181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[1181](info): set up 2 sockets
Sep 26 07:19:38 dist-test-slave-7zlw krb5kdc[1181](info): commencing operation
krb5kdc: starting...
W20260926 07:19:39.873304 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:39 dist-test-slave-7zlw krb5kdc[1181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:40.005612 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34681
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:34681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:40.128351 1194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:40.128664 1194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:40.128711 1194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:40.133206 1194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:40.133276 1194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:40.133297 1194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:40.133315 1194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:40.133370 1194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:40.138890 1194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:34681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:40.140079 1194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:40.141208 1194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:40.150156 1199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:40.150168 1202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:40.150168 1200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:40.150427 1194 server_base.cc:1034] running on GCE node
I20260926 07:19:40.151119 1194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:40.152380 1194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:40.153584 1194 hybrid_clock.cc:648] HybridClock initialized: now 1790407180153560 us; error 45 us; skew 500 ppm
Sep 26 07:19:40 dist-test-slave-7zlw krb5kdc[1181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:40.156665 1194 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:40.157684 1194 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:40.157787 1194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:40.158013 1194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:40.160249 1194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "b4825593d47641ab8ddee7c0418f3f96"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "a314b953ca2d5afb20fc9e3b52ad1648"
server_key_iv: "f4cff4b2902a325d20b7b169b6e95be5"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.160859 1194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "b4825593d47641ab8ddee7c0418f3f96"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "a314b953ca2d5afb20fc9e3b52ad1648"
server_key_iv: "f4cff4b2902a325d20b7b169b6e95be5"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.164939 1194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 07:19:40.167948 1208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:40.169414 1194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:40.169569 1194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "b4825593d47641ab8ddee7c0418f3f96"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "a314b953ca2d5afb20fc9e3b52ad1648"
server_key_iv: "f4cff4b2902a325d20b7b169b6e95be5"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.169715 1194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:40.180842 1194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:40.184073 1194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:40.184293 1194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:40.191488 1194 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:34681
I20260926 07:19:40.191771 1260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:34681 every 8 connection(s)
I20260926 07:19:40.192210 1194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:40.192931 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 1194
I20260926 07:19:40.193107 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:40.193393 29615 external_mini_cluster.cc:1428] Setting key 893e9379e00770d10ad6b41178873c62
I20260926 07:19:40.195720 1261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:40.202612 1261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96: Bootstrap starting.
Sep 26 07:19:40 dist-test-slave-7zlw krb5kdc[1181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:40.205577 1261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:40.206526 1261 log.cc:824] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:40.208715 1261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96: No bootstrap required, opened a new log
I20260926 07:19:40.209483 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.194870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.28.235.254:34681)
0926 07:19:40.195316 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:40.195335 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:40.195453 (+ 118us) server_negotiation.cc:407] Connection header received
0926 07:19:40.196344 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:40.196366 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:40.196787 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:40.197173 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:40.197860 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.199369 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.200359 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.201116 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.201267 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.204075 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:40.204083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:40.204101 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:40.204136 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:40.206314 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:40.206870 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:40.206877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:40.206883 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:40.207048 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:40.207360 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:40.207365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:40.207366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:40.207777 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:40.207981 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:40.208298 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 07:19:40.208598 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260926 07:19:40.212870 1261 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4825593d47641ab8ddee7c0418f3f96" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34681 } }
I20260926 07:19:40.213212 1261 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:40.213297 1261 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4825593d47641ab8ddee7c0418f3f96, State: Initialized, Role: FOLLOWER
I20260926 07:19:40.213747 1261 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4825593d47641ab8ddee7c0418f3f96" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34681 } }
I20260926 07:19:40.213894 1261 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:40.213981 1261 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:40.214095 1261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:40.215291 1261 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4825593d47641ab8ddee7c0418f3f96" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34681 } }
I20260926 07:19:40.215655 1261 leader_election.cc:302] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4825593d47641ab8ddee7c0418f3f96; no voters:
I20260926 07:19:40.215991 1261 leader_election.cc:288] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:40.216120 1266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:40.216418 1266 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [term 1 LEADER]: Becoming Leader. State: Replica: b4825593d47641ab8ddee7c0418f3f96, State: Running, Role: LEADER
I20260926 07:19:40.216759 1266 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4825593d47641ab8ddee7c0418f3f96" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34681 } }
I20260926 07:19:40.217563 1261 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:40.218794 1267 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4825593d47641ab8ddee7c0418f3f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4825593d47641ab8ddee7c0418f3f96" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34681 } } }
I20260926 07:19:40.218822 1268 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4825593d47641ab8ddee7c0418f3f96. Latest consensus state: current_term: 1 leader_uuid: "b4825593d47641ab8ddee7c0418f3f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4825593d47641ab8ddee7c0418f3f96" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34681 } } }
I20260926 07:19:40.218940 1268 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:40.218938 1267 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:40.219347 1275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:40.223203 1275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:40.228713 1275 catalog_manager.cc:1347] Generated new cluster ID: 99b7b448793844b5a7f6deadbef90c83
I20260926 07:19:40.228797 1275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:40.238565 1275 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:40.239885 1275 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:40.247932 1275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96: Generated new TSK 0
I20260926 07:19:40.248733 1275 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:19:40.285194 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34681
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:40.403963 1287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:40.404284 1287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:40.404381 1287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:40.408548 1287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:40.408646 1287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:40.408738 1287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:40.414312 1287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:40.415480 1287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:40.416539 1287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:40.426601 1293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:40.426664 1295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:40.426667 1292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:40.426882 1287 server_base.cc:1034] running on GCE node
I20260926 07:19:40.427277 1287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:40.427953 1287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:40.429158 1287 hybrid_clock.cc:648] HybridClock initialized: now 1790407180429140 us; error 31 us; skew 500 ppm
Sep 26 07:19:40 dist-test-slave-7zlw krb5kdc[1181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:40.432188 1287 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:19:40.433108 1287 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:40.433182 1287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:40.433379 1287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:40.435473 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "d9b73ec52e7c46a695c9be3771fae1be"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "e757310531805523c58b4eb87e57335a"
server_key_iv: "afd7d70cc50e8452bc75f1ca1a8892ad"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.436116 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "d9b73ec52e7c46a695c9be3771fae1be"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "e757310531805523c58b4eb87e57335a"
server_key_iv: "afd7d70cc50e8452bc75f1ca1a8892ad"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.440055 1287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:19:40.442694 1301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:40.443948 1287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:19:40.444104 1287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "d9b73ec52e7c46a695c9be3771fae1be"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "e757310531805523c58b4eb87e57335a"
server_key_iv: "afd7d70cc50e8452bc75f1ca1a8892ad"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.444226 1287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:40.465612 1287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:40.469259 1287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:40.469467 1287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:40.470093 1287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:40.470965 1287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:40.471037 1287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:40.471102 1287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:40.471145 1287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:40.480518 1287 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:32937
I20260926 07:19:40.480536 1414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:32937 every 8 connection(s)
I20260926 07:19:40.481210 1287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:40.482239 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 1287
I20260926 07:19:40.482367 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:40.482700 29615 external_mini_cluster.cc:1428] Setting key cd7d1b2f1baa7f09efa16492547d1970
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
Sep 26 07:19:40 dist-test-slave-7zlw krb5kdc[1181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:19:40.499188 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.483305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:52165 (local address 127.28.235.254:34681)
0926 07:19:40.483453 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:40.483458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:40.484459 (+ 1001us) server_negotiation.cc:407] Connection header received
0926 07:19:40.486044 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:40.486047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:40.486095 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:40.486165 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:40.487889 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.488526 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.490616 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.490867 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.490964 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.494306 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:40.494310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:40.494312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:40.494342 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:40.496569 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:40.497350 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:40.497353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:40.497355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:40.497462 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:40.497888 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:40.497892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:40.497894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:40.498117 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:40.498236 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:40.498842 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 07:19:40.498997 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:19:40.500113 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.483720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34681 (local address 127.28.235.193:52165)
0926 07:19:40.484284 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:40.484345 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:40.485774 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:40.486390 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:40.486402 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:40.486950 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:40.487687 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.487711 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.488695 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.488699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.490452 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.490467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.491064 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.491070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.491309 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.492016 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:40.492042 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:40.493995 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:40.496744 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:40.496753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:40.496774 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:40.497209 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:40.497583 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:40.497587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:40.497590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:40.497771 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:40.498252 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:40.498261 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:40.498557 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 07:19:40.498784 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 07:19:40.499179 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":119,"threads_started":1}
I20260926 07:19:40.501797 1415 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34681
I20260926 07:19:40.502100 1415 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:40.502728 1415 heartbeater.cc:511] Master 127.28.235.254:34681 requested a full tablet report, sending...
I20260926 07:19:40.504623 1225 ts_manager.cc:194] Registered new tserver with Master: d9b73ec52e7c46a695c9be3771fae1be (127.28.235.193:32937)
I20260926 07:19:40.505888 1225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:52165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:19:40.516391 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34681
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:40.629087 1420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:40.629387 1420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:40.629436 1420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:40.633636 1420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:40.633695 1420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:40.633757 1420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:40.638988 1420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:40.640024 1420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:40.640985 1420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:40.651124 1428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:40.651172 1426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:40.651203 1425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:40.651329 1420 server_base.cc:1034] running on GCE node
I20260926 07:19:40.651887 1420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:40.652608 1420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:40.653792 1420 hybrid_clock.cc:648] HybridClock initialized: now 1790407180653737 us; error 71 us; skew 500 ppm
Sep 26 07:19:40 dist-test-slave-7zlw krb5kdc[1181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:40.656657 1420 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:19:40.657508 1420 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:40.657605 1420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:40.657829 1420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:40.659921 1420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "9ba7c2796ff44d8192f31ce0eb327bcb"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "3cebd1bece36143a96d2aec6e9b0f01d"
server_key_iv: "5fc8a3c1eae74d195bd19fec14ce4cae"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.660490 1420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "9ba7c2796ff44d8192f31ce0eb327bcb"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "3cebd1bece36143a96d2aec6e9b0f01d"
server_key_iv: "5fc8a3c1eae74d195bd19fec14ce4cae"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.664438 1420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:19:40.667223 1434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:40.668596 1420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:40.668743 1420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "9ba7c2796ff44d8192f31ce0eb327bcb"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "3cebd1bece36143a96d2aec6e9b0f01d"
server_key_iv: "5fc8a3c1eae74d195bd19fec14ce4cae"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.668867 1420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:40.700924 1420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:40.704468 1420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:40.704682 1420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:40.705344 1420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:40.706231 1420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:40.706303 1420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:40.706377 1420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:40.706429 1420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:40.715975 1420 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:41477
I20260926 07:19:40.715991 1547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:41477 every 8 connection(s)
I20260926 07:19:40.716619 1420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:40.723917 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 1420
I20260926 07:19:40.724061 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:40.724380 29615 external_mini_cluster.cc:1428] Setting key 16c1fb94e41c3e10bcf884ecc39ada37
Sep 26 07:19:40 dist-test-slave-7zlw krb5kdc[1181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:40.732128 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.718554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:42321 (local address 127.28.235.254:34681)
0926 07:19:40.718699 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:40.718704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:40.719809 (+ 1105us) server_negotiation.cc:407] Connection header received
0926 07:19:40.721364 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:40.721367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:40.721419 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:40.721527 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:40.722740 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.723367 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.724729 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.724986 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.725181 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.728086 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:40.728090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:40.728092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:40.728116 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:40.729612 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:40.730315 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:40.730318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:40.730319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:40.730394 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:40.730820 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:40.730822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:40.730823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:40.730989 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:40.731061 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:40.731772 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 07:19:40.731873 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:19:40.733345 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.719057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34681 (local address 127.28.235.194:42321)
0926 07:19:40.719589 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:40.719677 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:40.721078 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:40.721698 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:40.721705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:40.722063 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:40.722576 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.722589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.723492 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.723494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.724608 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.724617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.725214 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.725218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.725418 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.726057 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:40.726074 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:40.727903 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:40.729790 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:40.729799 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:40.729821 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:40.730200 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:40.730487 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:40.730493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:40.730498 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:40.730716 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:40.731097 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:40.731108 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:40.731486 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 07:19:40.731744 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 07:19:40.732147 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":139,"threads_started":1}
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:19:40.734663 1548 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34681
I20260926 07:19:40.734912 1548 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:40.735396 1548 heartbeater.cc:511] Master 127.28.235.254:34681 requested a full tablet report, sending...
I20260926 07:19:40.736634 1225 ts_manager.cc:194] Registered new tserver with Master: 9ba7c2796ff44d8192f31ce0eb327bcb (127.28.235.194:41477)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
I20260926 07:19:40.737406 1225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:42321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:19:40.757447 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34681
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:40.872530 1553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:40.872854 1553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:40.872951 1553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:40.877267 1553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:40.877388 1553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:40.877487 1553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:40.883045 1553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:40.884205 1553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:40.885317 1553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:40.895325 1559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:40.895339 1558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:40.895370 1553 server_base.cc:1034] running on GCE node
W20260926 07:19:40.895576 1561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:40.896005 1553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:40.896672 1553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:40.897857 1553 hybrid_clock.cc:648] HybridClock initialized: now 1790407180897844 us; error 36 us; skew 500 ppm
Sep 26 07:19:40 dist-test-slave-7zlw krb5kdc[1181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:40.900718 1553 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:19:40.901546 1553 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:40.901652 1553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:40.901898 1553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:40.903985 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "649213685fda4516b61fa9e1b0c48bd4"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "6926fd2678e89654a3f0090b8f58ede1"
server_key_iv: "7a65d4f5001b20a3600666a759c47a71"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.904588 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "649213685fda4516b61fa9e1b0c48bd4"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "6926fd2678e89654a3f0090b8f58ede1"
server_key_iv: "7a65d4f5001b20a3600666a759c47a71"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.908598 1553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:19:40.911228 1567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:40.912431 1553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:19:40.912572 1553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "649213685fda4516b61fa9e1b0c48bd4"
format_stamp: "Formatted at 2026-09-26 07:19:40 on dist-test-slave-7zlw"
server_key: "6926fd2678e89654a3f0090b8f58ede1"
server_key_iv: "7a65d4f5001b20a3600666a759c47a71"
server_key_version: "encryptionkey@0"
I20260926 07:19:40.912714 1553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:40.924959 1553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:40.928066 1553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:40.928280 1553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:40.928892 1553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:40.929796 1553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:40.929870 1553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:40.929941 1553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:40.929983 1553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:40.939287 1553 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:33419
I20260926 07:19:40.939303 1680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:33419 every 8 connection(s)
I20260926 07:19:40.939941 1553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:40.944355 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 1553
I20260926 07:19:40.944504 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:40.944866 29615 external_mini_cluster.cc:1428] Setting key 430cd70c52c2bc7e89da2321a572c7cb
Sep 26 07:19:40 dist-test-slave-7zlw krb5kdc[1181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407180, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:40.954669 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.941723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:51133 (local address 127.28.235.254:34681)
0926 07:19:40.941884 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:40.941889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:40.943086 (+ 1197us) server_negotiation.cc:407] Connection header received
0926 07:19:40.944478 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:40.944482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:40.944545 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:40.944641 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:40.946157 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.946844 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.948197 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.948440 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.948526 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.950979 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:40.950982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:40.950984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:40.951009 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:40.952505 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:40.953141 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:40.953145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:40.953147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:40.953273 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:40.953639 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:40.953642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:40.953643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:40.953841 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:40.953923 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:40.954419 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 07:19:40.954534 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:19:40.955586 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.942399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34681 (local address 127.28.235.195:51133)
0926 07:19:40.942901 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:40.942945 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:40.944250 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:40.944826 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:40.944834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:40.945312 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:40.946002 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.946019 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.947015 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.947022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.948045 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.948054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.948580 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.948583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.948724 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.949330 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:40.949346 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:40.950823 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:40.952620 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:40.952625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:40.952638 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:40.952883 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:40.953378 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:40.953380 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:40.953382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:40.953517 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:40.953934 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:40.953940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:40.954195 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 07:19:40.954358 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 07:19:40.954649 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":117,"threads_started":1}
I20260926 07:19:40.956832 1681 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34681
I20260926 07:19:40.957226 1681 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:40.957731 1681 heartbeater.cc:511] Master 127.28.235.254:34681 requested a full tablet report, sending...
I20260926 07:19:40.958880 1225 ts_manager.cc:194] Registered new tserver with Master: 649213685fda4516b61fa9e1b0c48bd4 (127.28.235.195:33419)
I20260926 07:19:40.959422 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:19:40.959626 1225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:51133
I20260926 07:19:40.969331 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.961549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.28.235.254:34681)
0926 07:19:40.961673 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:40.961676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:40.962053 (+ 377us) server_negotiation.cc:407] Connection header received
0926 07:19:40.962124 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:40.962126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:40.962166 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:40.962238 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:40.962916 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.963421 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.964324 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.964560 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.964650 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.966196 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:40.966200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:40.966202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:40.966228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:40.967685 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:40.968100 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:40.968102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:40.968104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:40.968180 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:40.968418 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:40.968420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:40.968422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:40.968645 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:40.968750 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:40.969094 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 07:19:40.969209 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:19:40.972396 1225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:19:40.975836 1225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:40.990849 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.985090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:41477 (local address 127.0.0.1:41052)
0926 07:19:40.985654 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:40.985667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:40.985766 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:40.986392 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:40.986395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:40.986411 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:40.986501 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.986507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.987882 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.987885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.989173 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.989182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.990267 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.990269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.990379 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.990431 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:19:40.990553 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 07:19:40.990700 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":237,"thread_start_us":96,"threads_started":1}
I20260926 07:19:40.990841 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.984831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.28.235.194:41477)
0926 07:19:40.985364 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:40.985375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:40.985867 (+ 492us) server_negotiation.cc:407] Connection header received
0926 07:19:40.985948 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:40.985953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:40.986175 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:40.986319 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:40.986620 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.987705 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.989289 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.990149 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.990281 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.990434 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:40.990581 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 07:19:40.990682 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":148,"threads_started":1}
I20260926 07:19:40.990902 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.984641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.28.235.195:33419)
0926 07:19:40.984991 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:40.985003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:40.985742 (+ 739us) server_negotiation.cc:407] Connection header received
0926 07:19:40.985827 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:40.985834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:40.986053 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:40.986215 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:40.986516 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.987722 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.989492 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.990281 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.990390 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.990496 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:40.990670 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 07:19:40.990764 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":145,"threads_started":1}
I20260926 07:19:40.991282 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.984534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:33419 (local address 127.0.0.1:51430)
0926 07:19:40.985394 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:40.985445 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:40.985581 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:40.986218 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:40.986227 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:40.986256 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:40.986404 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.986411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.987838 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.987841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.989364 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.989374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.990387 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.990390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.990474 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.990503 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:19:40.990567 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:19:40.991166 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":606,"thread_start_us":97,"threads_started":1}
I20260926 07:19:40.992061 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.985332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:32937 (local address 127.0.0.1:41922)
0926 07:19:40.986110 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:40.986136 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:40.986217 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:40.986698 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:40.986701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:40.986717 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:40.986803 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.986809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.987741 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.987746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.989364 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:40.989374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.991652 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.991656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:40.991747 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.991771 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:19:40.991842 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:19:40.991915 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":196,"thread_start_us":66,"threads_started":1}
I20260926 07:19:40.992600 1697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:40.985662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.28.235.193:32937)
0926 07:19:40.986053 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:40.986059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:40.986308 (+ 249us) server_negotiation.cc:407] Connection header received
0926 07:19:40.986386 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:40.986391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:40.986563 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:40.986703 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:40.986894 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.987611 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.990556 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:40.991317 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:40.991437 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:40.991589 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:40.992342 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 07:19:40.992447 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":105,"threads_started":1}
I20260926 07:19:40.994839 1349 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3f3f2925d24c30b47cecda131da8f9 (DEFAULT_TABLE table=test-table [id=9cea9b2666324322b3d663862cc2b814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:40.994848 1482 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3f3f2925d24c30b47cecda131da8f9 (DEFAULT_TABLE table=test-table [id=9cea9b2666324322b3d663862cc2b814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:40.994740 1615 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3f3f2925d24c30b47cecda131da8f9 (DEFAULT_TABLE table=test-table [id=9cea9b2666324322b3d663862cc2b814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:40.995826 1349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3f3f2925d24c30b47cecda131da8f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:40.995848 1615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3f3f2925d24c30b47cecda131da8f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:40.995842 1482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3f3f2925d24c30b47cecda131da8f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:41.004737 1698 tablet_bootstrap.cc:492] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: Bootstrap starting.
I20260926 07:19:41.004818 1699 tablet_bootstrap.cc:492] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be: Bootstrap starting.
I20260926 07:19:41.006749 1698 tablet_bootstrap.cc:654] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:41.006989 1699 tablet_bootstrap.cc:654] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:41.007896 1700 tablet_bootstrap.cc:492] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: Bootstrap starting.
I20260926 07:19:41.007907 1699 log.cc:824] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:41.008031 1698 log.cc:824] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:41.010031 1699 tablet_bootstrap.cc:492] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be: No bootstrap required, opened a new log
I20260926 07:19:41.010066 1698 tablet_bootstrap.cc:492] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: No bootstrap required, opened a new log
I20260926 07:19:41.010254 1699 ts_tablet_manager.cc:1397] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:19:41.010255 1698 ts_tablet_manager.cc:1397] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:19:41.010388 1700 tablet_bootstrap.cc:654] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:41.011752 1700 log.cc:824] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:41.013967 1700 tablet_bootstrap.cc:492] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: No bootstrap required, opened a new log
I20260926 07:19:41.013936 1699 raft_consensus.cc:348] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 } } peers { permanent_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41477 } } peers { permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 } }
I20260926 07:19:41.014168 1700 ts_tablet_manager.cc:1397] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 07:19:41.014230 1699 raft_consensus.cc:374] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:41.014285 1699 raft_consensus.cc:729] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b73ec52e7c46a695c9be3771fae1be, State: Initialized, Role: FOLLOWER
I20260926 07:19:41.014274 1698 raft_consensus.cc:348] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 } } peers { permanent_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41477 } } peers { permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 } }
I20260926 07:19:41.014534 1698 raft_consensus.cc:374] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:41.014583 1698 raft_consensus.cc:729] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba7c2796ff44d8192f31ce0eb327bcb, State: Initialized, Role: FOLLOWER
I20260926 07:19:41.014773 1699 consensus_queue.cc:260] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 } } peers { permanent_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41477 } } peers { permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 } }
I20260926 07:19:41.015034 1698 consensus_queue.cc:260] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 } } peers { permanent_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41477 } } peers { permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 } }
I20260926 07:19:41.015528 1415 heartbeater.cc:503] Master 127.28.235.254:34681 was elected leader, sending a full tablet report...
I20260926 07:19:41.015813 1699 ts_tablet_manager.cc:1428] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 07:19:41.015934 1698 ts_tablet_manager.cc:1428] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:19:41.016279 1548 heartbeater.cc:503] Master 127.28.235.254:34681 was elected leader, sending a full tablet report...
I20260926 07:19:41.018100 1700 raft_consensus.cc:348] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 } } peers { permanent_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41477 } } peers { permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 } }
I20260926 07:19:41.018476 1700 raft_consensus.cc:374] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:41.018570 1700 raft_consensus.cc:729] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649213685fda4516b61fa9e1b0c48bd4, State: Initialized, Role: FOLLOWER
I20260926 07:19:41.019017 1700 consensus_queue.cc:260] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 } } peers { permanent_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41477 } } peers { permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 } }
I20260926 07:19:41.020011 1700 ts_tablet_manager.cc:1428] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:19:41.020391 1681 heartbeater.cc:503] Master 127.28.235.254:34681 was elected leader, sending a full tablet report...
I20260926 07:19:41.152694 1705 raft_consensus.cc:482] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:41.152897 1705 raft_consensus.cc:504] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 } } peers { permanent_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41477 } } peers { permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 } }
I20260926 07:19:41.154088 1705 leader_election.cc:288] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 649213685fda4516b61fa9e1b0c48bd4 (127.28.235.195:33419), d9b73ec52e7c46a695c9be3771fae1be (127.28.235.193:32937)
I20260926 07:19:41.159210 1697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.154483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:42057 (local address 127.28.235.193:32937)
0926 07:19:41.154613 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:41.154620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:41.154639 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:19:41.154832 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:41.154835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:41.154878 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:41.154936 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:41.155262 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:41.156076 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:41.157769 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:41.158403 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:41.158552 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:41.158609 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:41.158988 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 07:19:41.159085 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:19:41.159233 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.154611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:42635 (local address 127.28.235.195:33419)
0926 07:19:41.154775 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:41.154780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:41.154987 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:19:41.155124 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:41.155127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:41.155168 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:41.155266 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:41.155509 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:41.156234 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:41.157769 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:41.158397 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:41.158552 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:41.158609 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:41.158988 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 07:19:41.159131 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:19:41.159194 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.154387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:32937 (local address 127.28.235.194:42057)
0926 07:19:41.154559 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:41.154576 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:41.154686 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:41.154987 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:41.154990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:41.155002 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:41.155144 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:41.155149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:41.156194 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:41.156196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:41.157580 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:41.157593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:41.158537 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:41.158540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:41.158765 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:41.158811 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 07:19:41.158894 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:19:41.159008 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":94}
I20260926 07:19:41.159224 1707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.154387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:33419 (local address 127.28.235.194:42635)
0926 07:19:41.154861 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:41.154874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:41.155001 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:41.155295 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:41.155297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:41.155313 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:41.155402 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:41.155406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:41.156340 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:41.156343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:41.157615 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:41.157639 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:41.158537 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:41.158540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:41.158794 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:41.158815 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:19:41.158894 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:19:41.159008 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":263,"thread_start_us":132,"threads_started":1}
I20260926 07:19:41.160101 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3f3f2925d24c30b47cecda131da8f9" candidate_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b73ec52e7c46a695c9be3771fae1be" is_pre_election: true
I20260926 07:19:41.160144 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3f3f2925d24c30b47cecda131da8f9" candidate_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649213685fda4516b61fa9e1b0c48bd4" is_pre_election: true
I20260926 07:19:41.160451 1369 raft_consensus.cc:2454] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ba7c2796ff44d8192f31ce0eb327bcb in term 0.
I20260926 07:19:41.160451 1635 raft_consensus.cc:2454] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ba7c2796ff44d8192f31ce0eb327bcb in term 0.
I20260926 07:19:41.161002 1435 leader_election.cc:302] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 649213685fda4516b61fa9e1b0c48bd4, 9ba7c2796ff44d8192f31ce0eb327bcb, d9b73ec52e7c46a695c9be3771fae1be; no voters:
I20260926 07:19:41.161352 1705 raft_consensus.cc:2781] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:41.161460 1705 raft_consensus.cc:482] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:41.161522 1705 raft_consensus.cc:3037] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:41.162940 1705 raft_consensus.cc:504] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 } } peers { permanent_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41477 } } peers { permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 } }
I20260926 07:19:41.163470 1705 leader_election.cc:288] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [CANDIDATE]: Term 1 election: Requested vote from peers 649213685fda4516b61fa9e1b0c48bd4 (127.28.235.195:33419), d9b73ec52e7c46a695c9be3771fae1be (127.28.235.193:32937)
I20260926 07:19:41.164081 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3f3f2925d24c30b47cecda131da8f9" candidate_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b73ec52e7c46a695c9be3771fae1be"
I20260926 07:19:41.164149 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3f3f2925d24c30b47cecda131da8f9" candidate_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649213685fda4516b61fa9e1b0c48bd4"
I20260926 07:19:41.164225 1369 raft_consensus.cc:3037] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:41.164366 1635 raft_consensus.cc:3037] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:41.165656 1369 raft_consensus.cc:2454] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ba7c2796ff44d8192f31ce0eb327bcb in term 1.
I20260926 07:19:41.165663 1635 raft_consensus.cc:2454] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ba7c2796ff44d8192f31ce0eb327bcb in term 1.
I20260926 07:19:41.166105 1436 leader_election.cc:302] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ba7c2796ff44d8192f31ce0eb327bcb, d9b73ec52e7c46a695c9be3771fae1be; no voters:
I20260926 07:19:41.166350 1705 raft_consensus.cc:2781] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:41.166599 1705 raft_consensus.cc:686] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 1 LEADER]: Becoming Leader. State: Replica: 9ba7c2796ff44d8192f31ce0eb327bcb, State: Running, Role: LEADER
I20260926 07:19:41.166956 1705 consensus_queue.cc:237] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 } } peers { permanent_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41477 } } peers { permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 } }
I20260926 07:19:41.170516 1224 catalog_manager.cc:5495] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ba7c2796ff44d8192f31ce0eb327bcb (127.28.235.194). New cstate: current_term: 1 leader_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ba7c2796ff44d8192f31ce0eb327bcb" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:41.186873 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.183031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.28.235.194:41477)
0926 07:19:41.183196 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:41.183200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:41.183216 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:19:41.183401 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:41.183404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:41.183502 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:41.183593 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:41.183842 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:41.184382 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:41.185260 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:41.185489 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:41.185572 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:41.185954 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:41.186396 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:41.186505 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:41.186611 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 07:19:41.186662 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 07:19:41.190814 1682 tablet.cc:2369] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:19:41.193228 1635 raft_consensus.cc:1261] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4 [term 1 FOLLOWER]: Refusing update from remote peer 9ba7c2796ff44d8192f31ce0eb327bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:41.193262 1369 raft_consensus.cc:1261] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be [term 1 FOLLOWER]: Refusing update from remote peer 9ba7c2796ff44d8192f31ce0eb327bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:41.193948 1705 consensus_queue.cc:1035] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649213685fda4516b61fa9e1b0c48bd4" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:41.194182 1708 consensus_queue.cc:1035] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b73ec52e7c46a695c9be3771fae1be" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 32937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:41.214725 1224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:41.214926 1224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:41.217936 1224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4825593d47641ab8ddee7c0418f3f96: Sending DeleteTablet for 3 replicas of tablet 7e3f3f2925d24c30b47cecda131da8f9
I20260926 07:19:41.218907 1482 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e3f3f2925d24c30b47cecda131da8f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:41 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:41052
I20260926 07:19:41.218932 1615 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e3f3f2925d24c30b47cecda131da8f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:41 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:51430
I20260926 07:19:41.219592 1721 tablet_replica.cc:330] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: stopping tablet replica
I20260926 07:19:41.219764 1349 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e3f3f2925d24c30b47cecda131da8f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:41 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:41922
I20260926 07:19:41.219897 1721 raft_consensus.cc:2229] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:41.220218 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 1287
I20260926 07:19:41.220285 1721 raft_consensus.cc:2258] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:41.220295 1723 tablet_replica.cc:330] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be: stopping tablet replica
I20260926 07:19:41.220650 1723 raft_consensus.cc:2229] T 7e3f3f2925d24c30b47cecda131da8f9 P d9b73ec52e7c46a695c9be3771fae1be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:41.223426 1721 ts_tablet_manager.cc:1905] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:41.226257 1721 ts_tablet_manager.cc:1918] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:41.226357 1721 log.cc:1196] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/7e3f3f2925d24c30b47cecda131da8f9
I20260926 07:19:41.226667 1721 ts_tablet_manager.cc:1939] T 7e3f3f2925d24c30b47cecda131da8f9 P 9ba7c2796ff44d8192f31ce0eb327bcb: Deleting consensus metadata
I20260926 07:19:41.227975 1722 tablet_replica.cc:330] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: stopping tablet replica
I20260926 07:19:41.228029 1212 catalog_manager.cc:4841] TS 9ba7c2796ff44d8192f31ce0eb327bcb (127.28.235.194:41477): tablet 7e3f3f2925d24c30b47cecda131da8f9 (table test-table [id=9cea9b2666324322b3d663862cc2b814]) successfully deleted
I20260926 07:19:41.228309 1722 raft_consensus.cc:2229] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:41.228647 1722 raft_consensus.cc:2258] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:41.230321 1722 ts_tablet_manager.cc:1905] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:19:41.231019 1210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:32937 (error 108)
I20260926 07:19:41.232245 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.231876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:32937 (local address 127.0.0.1:41932)
0926 07:19:41.232031 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:41.232123 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 07:19:41.232381 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 1420
I20260926 07:19:41.233045 1722 ts_tablet_manager.cc:1918] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:41.233130 1722 log.cc:1196] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/7e3f3f2925d24c30b47cecda131da8f9
I20260926 07:19:41.233440 1722 ts_tablet_manager.cc:1939] T 7e3f3f2925d24c30b47cecda131da8f9 P 649213685fda4516b61fa9e1b0c48bd4: Deleting consensus metadata
W20260926 07:19:41.233773 1210 catalog_manager.cc:4584] TS d9b73ec52e7c46a695c9be3771fae1be (127.28.235.193:32937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e3f3f2925d24c30b47cecda131da8f9: Network error: Client connection negotiation failed: client connection to 127.28.235.193:32937: connect: Connection refused (error 111)
I20260926 07:19:41.234881 1209 catalog_manager.cc:4841] TS 649213685fda4516b61fa9e1b0c48bd4 (127.28.235.195:33419): tablet 7e3f3f2925d24c30b47cecda131da8f9 (table test-table [id=9cea9b2666324322b3d663862cc2b814]) successfully deleted
I20260926 07:19:41.239305 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 1553
I20260926 07:19:41.244805 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 1194
2026-09-26T07:19:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2434 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T07:19:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:41Z Disabled control of system clock
I20260926 07:19:41.275470 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:38571
--webserver_interface=127.28.235.254
--webserver_port=0
--builtin_ntp_servers=127.28.235.212:36645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:19:41.390864 1728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:41.391163 1728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:41.391203 1728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:41.395370 1728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:41.395434 1728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:41.395449 1728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:41.395466 1728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:41.395483 1728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:41.400398 1728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:38571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:38571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:41.401605 1728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:41.403043 1728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:41.411762 1733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:41.411736 1736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:41.411846 1734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:41.411952 1728 server_base.cc:1034] running on GCE node
I20260926 07:19:41.412503 1728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:41.413704 1728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:41.414892 1728 hybrid_clock.cc:648] HybridClock initialized: now 1790407181414876 us; error 34 us; skew 500 ppm
I20260926 07:19:41.416779 1728 webserver.cc:460] Webserver started at http://127.28.235.254:40239/ using document root <none> and password file <none>
I20260926 07:19:41.417418 1728 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:41.417507 1728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:41.417745 1728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:41.419745 1728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "72481352d1ee48d69e50c57274cc3960"
format_stamp: "Formatted at 2026-09-26 07:19:41 on dist-test-slave-7zlw"
server_key: "298bb2189ed583aa473dc9e7d1d225fa"
server_key_iv: "da96fa661c3703678d7738656b39998f"
server_key_version: "encryptionkey@0"
I20260926 07:19:41.420291 1728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "72481352d1ee48d69e50c57274cc3960"
format_stamp: "Formatted at 2026-09-26 07:19:41 on dist-test-slave-7zlw"
server_key: "298bb2189ed583aa473dc9e7d1d225fa"
server_key_iv: "da96fa661c3703678d7738656b39998f"
server_key_version: "encryptionkey@0"
I20260926 07:19:41.424064 1728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:19:41.426592 1742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:41.427910 1728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:19:41.428049 1728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "72481352d1ee48d69e50c57274cc3960"
format_stamp: "Formatted at 2026-09-26 07:19:41 on dist-test-slave-7zlw"
server_key: "298bb2189ed583aa473dc9e7d1d225fa"
server_key_iv: "da96fa661c3703678d7738656b39998f"
server_key_version: "encryptionkey@0"
I20260926 07:19:41.428150 1728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:41.432765 1728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:41.433429 1728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:41.433606 1728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:41.440706 1728 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:38571
I20260926 07:19:41.440716 1794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:38571 every 8 connection(s)
I20260926 07:19:41.441675 1728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:41.445228 1795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:41.451577 1795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960: Bootstrap starting.
I20260926 07:19:41.451864 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 1728
I20260926 07:19:41.451992 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:41.452306 29615 external_mini_cluster.cc:1428] Setting key 03a19832b4ffa9806d17e3cdfbf80fd0
I20260926 07:19:41.454901 1795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:41.455727 1795 log.cc:824] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:41.458314 1795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960: No bootstrap required, opened a new log
I20260926 07:19:41.458523 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.453945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.28.235.254:38571)
0926 07:19:41.454652 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:41.454668 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:41.454705 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:19:41.455495 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:41.455515 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:41.455955 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:41.456308 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:41.457182 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:41.457186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:41.457199 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:41.457209 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:41.457265 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:41.457281 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:41.457405 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:41.457506 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 07:19:41.457772 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":143,"threads_started":1}
I20260926 07:19:41.462430 1795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72481352d1ee48d69e50c57274cc3960" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38571 } }
I20260926 07:19:41.462667 1795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:41.462742 1795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72481352d1ee48d69e50c57274cc3960, State: Initialized, Role: FOLLOWER
I20260926 07:19:41.463186 1795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72481352d1ee48d69e50c57274cc3960" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38571 } }
I20260926 07:19:41.463351 1795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:41.463452 1795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:41.463587 1795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:41.464624 1795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72481352d1ee48d69e50c57274cc3960" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38571 } }
I20260926 07:19:41.465070 1795 leader_election.cc:302] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72481352d1ee48d69e50c57274cc3960; no voters:
I20260926 07:19:41.465417 1795 leader_election.cc:288] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:41.465538 1800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:41.465795 1800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [term 1 LEADER]: Becoming Leader. State: Replica: 72481352d1ee48d69e50c57274cc3960, State: Running, Role: LEADER
I20260926 07:19:41.466092 1800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72481352d1ee48d69e50c57274cc3960" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38571 } }
I20260926 07:19:41.466787 1795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:41.467869 1801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72481352d1ee48d69e50c57274cc3960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72481352d1ee48d69e50c57274cc3960" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38571 } } }
I20260926 07:19:41.468029 1801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:41.468405 1809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:41.468405 1802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72481352d1ee48d69e50c57274cc3960. Latest consensus state: current_term: 1 leader_uuid: "72481352d1ee48d69e50c57274cc3960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72481352d1ee48d69e50c57274cc3960" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38571 } } }
I20260926 07:19:41.468519 1802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:41.471513 1809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:41.476889 1809 catalog_manager.cc:1347] Generated new cluster ID: 0ed5a5f1da964ac38c5010258f905c40
I20260926 07:19:41.476998 1809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:41.496109 1809 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:41.496469 1809 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:41.506524 1809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960: Generated new TSK 0
I20260926 07:19:41.507426 1809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:19:41.514333 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38571
--builtin_ntp_servers=127.28.235.212:36645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:19:41.626251 1819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:41.626545 1819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:41.626616 1819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:41.630604 1819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:41.630687 1819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:41.630785 1819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:41.635865 1819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:41.636955 1819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:41.638351 1819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:41.647998 1824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:41.648312 1819 server_base.cc:1034] running on GCE node
W20260926 07:19:41.648125 1827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:41.647998 1825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:41.648778 1819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:41.649477 1819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:41.650674 1819 hybrid_clock.cc:648] HybridClock initialized: now 1790407181650637 us; error 54 us; skew 500 ppm
I20260926 07:19:41.652784 1819 webserver.cc:460] Webserver started at http://127.28.235.193:45417/ using document root <none> and password file <none>
I20260926 07:19:41.653548 1819 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:41.653657 1819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:41.653924 1819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:41.656205 1819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "6f3202a5abb54e0fa6a556d7334d33e6"
format_stamp: "Formatted at 2026-09-26 07:19:41 on dist-test-slave-7zlw"
server_key: "4697e3d95f900b585101d4b5553d9a69"
server_key_iv: "44687b732aabca3b6077385e7c340b96"
server_key_version: "encryptionkey@0"
I20260926 07:19:41.656889 1819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "6f3202a5abb54e0fa6a556d7334d33e6"
format_stamp: "Formatted at 2026-09-26 07:19:41 on dist-test-slave-7zlw"
server_key: "4697e3d95f900b585101d4b5553d9a69"
server_key_iv: "44687b732aabca3b6077385e7c340b96"
server_key_version: "encryptionkey@0"
I20260926 07:19:41.661646 1819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:19:41.664155 1833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:41.665480 1819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:19:41.665620 1819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "6f3202a5abb54e0fa6a556d7334d33e6"
format_stamp: "Formatted at 2026-09-26 07:19:41 on dist-test-slave-7zlw"
server_key: "4697e3d95f900b585101d4b5553d9a69"
server_key_iv: "44687b732aabca3b6077385e7c340b96"
server_key_version: "encryptionkey@0"
I20260926 07:19:41.665738 1819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:41.670691 1819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:41.671308 1819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:41.671499 1819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:41.672052 1819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:41.672981 1819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:41.673071 1819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:41.673162 1819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:41.673213 1819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:41.681630 1819 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:38335
I20260926 07:19:41.681694 1946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:38335 every 8 connection(s)
I20260926 07:19:41.682569 1819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:41.688660 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.684401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:48953 (local address 127.28.235.254:38571)
0926 07:19:41.684590 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:41.684593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:41.685735 (+ 1142us) server_negotiation.cc:407] Connection header received
0926 07:19:41.686954 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:41.686956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:41.686996 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:41.687090 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:41.688242 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:41.688245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:41.688247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:41.688260 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:41.688289 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:41.688298 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:41.688384 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:41.688517 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 07:19:41.688553 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 07:19:41.689600 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.684796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:38571 (local address 127.28.235.193:48953)
0926 07:19:41.685529 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:41.685580 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:41.686757 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:41.687273 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:41.687280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:41.687369 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:41.687973 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:19:41.687974 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:41.688014 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:19:41.688082 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:41.688382 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:41.688389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:41.688393 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:19:41.688550 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 07:19:41.688742 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":135,"threads_started":1}
I20260926 07:19:41.690830 1947 heartbeater.cc:347] Connected to a master server at 127.28.235.254:38571
I20260926 07:19:41.691078 1947 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:41.691097 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 1819
I20260926 07:19:41.691274 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:41.691565 29615 external_mini_cluster.cc:1428] Setting key 6cbdc9f375ba21727b2bfe9f7f17b043
I20260926 07:19:41.691573 1947 heartbeater.cc:511] Master 127.28.235.254:38571 requested a full tablet report, sending...
I20260926 07:19:41.693060 1758 ts_manager.cc:194] Registered new tserver with Master: 6f3202a5abb54e0fa6a556d7334d33e6 (127.28.235.193:38335)
I20260926 07:19:41.693540 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38571
--builtin_ntp_servers=127.28.235.212:36645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:19:41.806806 1950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:41.807140 1950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:41.807214 1950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:41.811542 1950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:41.811641 1950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:41.811733 1950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:41.816640 1950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:41.817878 1950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:41.819274 1950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:41.829202 1958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:41.829185 1955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:41.829185 1956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:41.829526 1950 server_base.cc:1034] running on GCE node
I20260926 07:19:41.829991 1950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:41.830623 1950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:41.831781 1950 hybrid_clock.cc:648] HybridClock initialized: now 1790407181831763 us; error 32 us; skew 500 ppm
I20260926 07:19:41.833804 1950 webserver.cc:460] Webserver started at http://127.28.235.194:46765/ using document root <none> and password file <none>
I20260926 07:19:41.834424 1950 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:41.834487 1950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:41.834702 1950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:41.836903 1950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "6b749b1e17224146ba2433db4a8e66ff"
format_stamp: "Formatted at 2026-09-26 07:19:41 on dist-test-slave-7zlw"
server_key: "f8cb4173d68699ebdbfc482efa9874c3"
server_key_iv: "13fa6fb31a8702bfc8038283f622cde6"
server_key_version: "encryptionkey@0"
I20260926 07:19:41.837540 1950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "6b749b1e17224146ba2433db4a8e66ff"
format_stamp: "Formatted at 2026-09-26 07:19:41 on dist-test-slave-7zlw"
server_key: "f8cb4173d68699ebdbfc482efa9874c3"
server_key_iv: "13fa6fb31a8702bfc8038283f622cde6"
server_key_version: "encryptionkey@0"
I20260926 07:19:41.841478 1950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:41.844722 1964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:41.846280 1950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:41.846400 1950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "6b749b1e17224146ba2433db4a8e66ff"
format_stamp: "Formatted at 2026-09-26 07:19:41 on dist-test-slave-7zlw"
server_key: "f8cb4173d68699ebdbfc482efa9874c3"
server_key_iv: "13fa6fb31a8702bfc8038283f622cde6"
server_key_version: "encryptionkey@0"
I20260926 07:19:41.846508 1950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:41.851431 1950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:41.852048 1950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:41.852195 1950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:41.852728 1950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:41.853762 1950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:41.853816 1950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:41.853857 1950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:41.853874 1950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:41.863484 1950 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:40231
I20260926 07:19:41.863497 2077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:40231 every 8 connection(s)
I20260926 07:19:41.864521 1950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:41.870141 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 1950
I20260926 07:19:41.870296 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:41.870587 29615 external_mini_cluster.cc:1428] Setting key d2e16b59fcacb3c1f1d66204d0b25ee9
I20260926 07:19:41.872231 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.866636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:53957 (local address 127.28.235.254:38571)
0926 07:19:41.866808 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:41.866812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:41.868248 (+ 1436us) server_negotiation.cc:407] Connection header received
0926 07:19:41.870046 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:41.870077 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:41.870135 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:41.870250 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:41.871718 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:41.871723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:41.871725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:41.871740 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:41.871772 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:41.871781 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:41.871879 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:41.872047 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 07:19:41.872096 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:19:41.872920 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38571
--builtin_ntp_servers=127.28.235.212:36645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 07:19:41.873330 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:41.867091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:38571 (local address 127.28.235.194:53957)
0926 07:19:41.868075 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:41.868123 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:41.869681 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:41.870515 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:41.870523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:41.870622 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:41.871376 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:19:41.871378 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:41.871423 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:19:41.871526 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:41.871876 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:41.871886 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:41.871895 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:19:41.872100 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 07:19:41.872395 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":118,"threads_started":1}
I20260926 07:19:41.874734 2078 heartbeater.cc:347] Connected to a master server at 127.28.235.254:38571
I20260926 07:19:41.874981 2078 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:41.875415 2078 heartbeater.cc:511] Master 127.28.235.254:38571 requested a full tablet report, sending...
I20260926 07:19:41.876513 1758 ts_manager.cc:194] Registered new tserver with Master: 6b749b1e17224146ba2433db4a8e66ff (127.28.235.194:40231)
W20260926 07:19:41.986972 2081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:41.987453 2081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:41.987543 2081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:41.991627 2081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:41.991683 2081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:41.991741 2081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:41.996438 2081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:41.997598 2081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:41.998924 2081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:19:42.009091 2081 server_base.cc:1034] running on GCE node
W20260926 07:19:42.008919 2086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:42.008973 2087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:42.008937 2089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:42.009786 2081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:42.010530 2081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:42.011714 2081 hybrid_clock.cc:648] HybridClock initialized: now 1790407182011697 us; error 32 us; skew 500 ppm
I20260926 07:19:42.013592 2081 webserver.cc:460] Webserver started at http://127.28.235.195:43393/ using document root <none> and password file <none>
I20260926 07:19:42.014195 2081 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:42.014287 2081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:42.014549 2081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:42.016582 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "10038123eaf144369edb79b5c85de258"
format_stamp: "Formatted at 2026-09-26 07:19:42 on dist-test-slave-7zlw"
server_key: "94cbf6118015c1976586f47f407d263d"
server_key_iv: "f6b68770a106d51e809b29768553ed28"
server_key_version: "encryptionkey@0"
I20260926 07:19:42.017216 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "10038123eaf144369edb79b5c85de258"
format_stamp: "Formatted at 2026-09-26 07:19:42 on dist-test-slave-7zlw"
server_key: "94cbf6118015c1976586f47f407d263d"
server_key_iv: "f6b68770a106d51e809b29768553ed28"
server_key_version: "encryptionkey@0"
I20260926 07:19:42.021061 2081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:42.024375 2095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:42.025838 2081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:42.025938 2081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "10038123eaf144369edb79b5c85de258"
format_stamp: "Formatted at 2026-09-26 07:19:42 on dist-test-slave-7zlw"
server_key: "94cbf6118015c1976586f47f407d263d"
server_key_iv: "f6b68770a106d51e809b29768553ed28"
server_key_version: "encryptionkey@0"
I20260926 07:19:42.026047 2081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:42.030642 2081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:42.031234 2081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:42.031409 2081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:42.031929 2081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:42.032814 2081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:42.032881 2081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:42.032945 2081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:42.032981 2081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:42.042009 2081 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:35843
I20260926 07:19:42.042075 2208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:35843 every 8 connection(s)
I20260926 07:19:42.042932 2081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:42.049119 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.044772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:36007 (local address 127.28.235.254:38571)
0926 07:19:42.044905 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.044912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.045986 (+ 1074us) server_negotiation.cc:407] Connection header received
0926 07:19:42.047307 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.047311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.047360 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.047442 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:42.048601 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:42.048604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:42.048607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:42.048620 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:42.048647 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:42.048661 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:42.048768 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:42.048902 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 07:19:42.048951 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:19:42.049847 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 2081
I20260926 07:19:42.049965 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:42.050182 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.045207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:38571 (local address 127.28.235.195:36007)
0926 07:19:42.045842 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:42.045881 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:42.047086 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:42.047609 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:42.047616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:42.047716 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:42.048336 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:19:42.048337 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:42.048378 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:19:42.048445 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:42.048747 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:42.048755 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:42.048760 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:19:42.048948 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 07:19:42.049226 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260926 07:19:42.050275 29615 external_mini_cluster.cc:1428] Setting key bee1dc3baa3febbd4facde556a570c17
I20260926 07:19:42.051568 2209 heartbeater.cc:347] Connected to a master server at 127.28.235.254:38571
I20260926 07:19:42.051810 2209 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:42.052234 2209 heartbeater.cc:511] Master 127.28.235.254:38571 requested a full tablet report, sending...
I20260926 07:19:42.053321 1759 ts_manager.cc:194] Registered new tserver with Master: 10038123eaf144369edb79b5c85de258 (127.28.235.195:35843)
I20260926 07:19:42.064380 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:19:42.068763 1798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.066573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.28.235.254:38571)
0926 07:19:42.066737 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.066741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.067010 (+ 269us) server_negotiation.cc:407] Connection header received
0926 07:19:42.067219 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.067222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.067302 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.067396 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:42.068387 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:42.068390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:42.068392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:42.068403 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:42.068421 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:42.068429 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:42.068500 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:42.068555 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 07:19:42.068592 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:19:42.072079 1759 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:19:42.075301 1759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:42.087338 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.084345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:35843 (local address 127.0.0.1:58968)
0926 07:19:42.084980 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:42.084992 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:42.085072 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:42.085791 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:42.085797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:42.085877 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:42.086565 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:19:42.086567 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:42.086587 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:19:42.086655 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:42.086965 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:42.086976 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:42.086982 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:19:42.087091 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:19:42.087158 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":80,"threads_started":1}
I20260926 07:19:42.087400 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.084171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.28.235.193:38335)
0926 07:19:42.085067 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.085087 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.085114 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:19:42.085180 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.085185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.085387 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.085529 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:42.086885 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:42.086893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:42.086905 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:42.086920 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:42.086961 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:42.086996 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:42.087131 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:42.087190 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:19:42.087263 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":84,"threads_started":1}
I20260926 07:19:42.087656 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.084895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.28.235.195:35843)
0926 07:19:42.085360 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.085370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.085392 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:19:42.085445 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.085454 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.085617 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.086172 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:42.086770 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:42.086775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:42.086786 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:42.086796 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:42.086839 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:42.086864 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:42.086971 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:42.087288 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 07:19:42.087364 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":184,"threads_started":1}
I20260926 07:19:42.088079 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.084103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:40231 (local address 127.0.0.1:35922)
0926 07:19:42.084656 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:42.084707 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:42.084810 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:42.085936 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:42.085939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:42.085955 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:42.086839 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:19:42.086842 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:42.086853 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:19:42.086873 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:42.087828 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:42.087831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:42.087834 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:19:42.087923 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:19:42.087967 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":79,"threads_started":1}
I20260926 07:19:42.088150 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.085213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.28.235.194:40231)
0926 07:19:42.085510 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.085526 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.085551 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:19:42.085607 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.085612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.085785 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.087032 (+ 1247us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:42.087622 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:42.087627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:42.087638 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:42.087649 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:42.087685 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:42.087711 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:42.087852 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:42.087952 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 07:19:42.088013 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":120,"threads_started":1}
I20260926 07:19:42.088800 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.084115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38335 (local address 127.0.0.1:39710)
0926 07:19:42.084896 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:42.084909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:42.084983 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:42.086230 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:42.086234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:42.086249 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:42.086704 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:19:42.086707 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:42.086719 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:19:42.086748 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:42.087096 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:42.087100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:42.087103 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:19:42.087167 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:19:42.087204 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":110,"threads_started":1}
I20260926 07:19:42.090583 2143 tablet_service.cc:1467] Processing CreateTablet for tablet 4875829129374bbfa2f686cfee7e0495 (DEFAULT_TABLE table=test-table [id=c1c5f4d0b1d1468fa16d1d3b4b56ac6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:42.091555 2143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4875829129374bbfa2f686cfee7e0495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:42.091274 1881 tablet_service.cc:1467] Processing CreateTablet for tablet 4875829129374bbfa2f686cfee7e0495 (DEFAULT_TABLE table=test-table [id=c1c5f4d0b1d1468fa16d1d3b4b56ac6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:42.091271 2012 tablet_service.cc:1467] Processing CreateTablet for tablet 4875829129374bbfa2f686cfee7e0495 (DEFAULT_TABLE table=test-table [id=c1c5f4d0b1d1468fa16d1d3b4b56ac6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:42.092278 2012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4875829129374bbfa2f686cfee7e0495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:42.092278 1881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4875829129374bbfa2f686cfee7e0495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:42.097890 2226 tablet_bootstrap.cc:492] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: Bootstrap starting.
I20260926 07:19:42.099653 2227 tablet_bootstrap.cc:492] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6: Bootstrap starting.
I20260926 07:19:42.100225 2226 tablet_bootstrap.cc:654] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:42.101150 2226 log.cc:824] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:42.101758 2227 tablet_bootstrap.cc:654] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:42.101805 2228 tablet_bootstrap.cc:492] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff: Bootstrap starting.
I20260926 07:19:42.102583 2227 log.cc:824] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:42.103058 2226 tablet_bootstrap.cc:492] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: No bootstrap required, opened a new log
I20260926 07:19:42.103296 2226 ts_tablet_manager.cc:1397] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:19:42.103997 2228 tablet_bootstrap.cc:654] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:42.104696 2227 tablet_bootstrap.cc:492] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6: No bootstrap required, opened a new log
I20260926 07:19:42.104781 2228 log.cc:824] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:42.104884 2227 ts_tablet_manager.cc:1397] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:19:42.106814 2228 tablet_bootstrap.cc:492] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff: No bootstrap required, opened a new log
I20260926 07:19:42.107009 2228 ts_tablet_manager.cc:1397] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:19:42.107332 2226 raft_consensus.cc:348] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38335 } } peers { permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 } } peers { permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 } }
I20260926 07:19:42.107642 2226 raft_consensus.cc:374] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:42.107699 2226 raft_consensus.cc:729] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10038123eaf144369edb79b5c85de258, State: Initialized, Role: FOLLOWER
I20260926 07:19:42.108165 2226 consensus_queue.cc:260] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38335 } } peers { permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 } } peers { permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 } }
I20260926 07:19:42.108772 2209 heartbeater.cc:503] Master 127.28.235.254:38571 was elected leader, sending a full tablet report...
I20260926 07:19:42.109190 2226 ts_tablet_manager.cc:1428] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:19:42.109388 2227 raft_consensus.cc:348] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38335 } } peers { permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 } } peers { permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 } }
I20260926 07:19:42.109650 2227 raft_consensus.cc:374] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:42.109735 2227 raft_consensus.cc:729] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3202a5abb54e0fa6a556d7334d33e6, State: Initialized, Role: FOLLOWER
I20260926 07:19:42.110289 2227 consensus_queue.cc:260] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38335 } } peers { permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 } } peers { permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 } }
I20260926 07:19:42.111249 2227 ts_tablet_manager.cc:1428] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:19:42.111528 2228 raft_consensus.cc:348] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38335 } } peers { permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 } } peers { permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 } }
I20260926 07:19:42.111732 2228 raft_consensus.cc:374] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:42.111822 2228 raft_consensus.cc:729] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b749b1e17224146ba2433db4a8e66ff, State: Initialized, Role: FOLLOWER
I20260926 07:19:42.111893 1947 heartbeater.cc:503] Master 127.28.235.254:38571 was elected leader, sending a full tablet report...
I20260926 07:19:42.112268 2228 consensus_queue.cc:260] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38335 } } peers { permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 } } peers { permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 } }
I20260926 07:19:42.112829 2078 heartbeater.cc:503] Master 127.28.235.254:38571 was elected leader, sending a full tablet report...
I20260926 07:19:42.113576 2228 ts_tablet_manager.cc:1428] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260926 07:19:42.115557 2079 tablet.cc:2369] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:19:42.183769 1948 tablet.cc:2369] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:19:42.293941 2210 tablet.cc:2369] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:19:42.340065 2233 raft_consensus.cc:482] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:42.340358 2233 raft_consensus.cc:504] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38335 } } peers { permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 } } peers { permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 } }
I20260926 07:19:42.341710 2233 leader_election.cc:288] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b749b1e17224146ba2433db4a8e66ff (127.28.235.194:40231), 10038123eaf144369edb79b5c85de258 (127.28.235.195:35843)
I20260926 07:19:42.344070 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.342101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:36099 (local address 127.28.235.194:40231)
0926 07:19:42.342455 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.342462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.342480 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:19:42.342561 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.342564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.342604 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.342689 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:42.343581 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:42.343585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:42.343588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:42.343601 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:42.343622 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:42.343632 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:42.343726 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:42.343890 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 07:19:42.343957 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260926 07:19:42.344070 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.342162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:52461 (local address 127.28.235.195:35843)
0926 07:19:42.342299 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.342304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.342440 (+ 136us) server_negotiation.cc:407] Connection header received
0926 07:19:42.342561 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.342564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.342604 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.342689 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:42.343581 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:42.343585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:42.343588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:42.343601 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:42.343622 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:42.343632 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:42.343726 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:42.343890 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 07:19:42.343933 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:19:42.344108 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.342010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:35843 (local address 127.28.235.193:52461)
0926 07:19:42.342299 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:42.342328 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:42.342444 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:42.342768 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:42.342775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:42.342792 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:42.343328 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:19:42.343330 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:42.343370 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:19:42.343409 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:42.343773 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:42.343775 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:42.343778 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:19:42.343880 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:19:42.343950 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":21,"thread_start_us":73,"threads_started":1}
I20260926 07:19:42.344111 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.342012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:40231 (local address 127.28.235.193:36099)
0926 07:19:42.342397 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:42.342409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:42.342462 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:42.342768 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:42.342772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:42.342792 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:42.343423 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:19:42.343425 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:42.343435 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:19:42.343460 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:42.343770 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:42.343774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:42.343778 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:19:42.343880 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:19:42.343993 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":54,"thread_start_us":102,"threads_started":1}
I20260926 07:19:42.344924 2032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4875829129374bbfa2f686cfee7e0495" candidate_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b749b1e17224146ba2433db4a8e66ff" is_pre_election: true
I20260926 07:19:42.344921 2163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4875829129374bbfa2f686cfee7e0495" candidate_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10038123eaf144369edb79b5c85de258" is_pre_election: true
I20260926 07:19:42.345287 2163 raft_consensus.cc:2454] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f3202a5abb54e0fa6a556d7334d33e6 in term 0.
I20260926 07:19:42.345287 2032 raft_consensus.cc:2454] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f3202a5abb54e0fa6a556d7334d33e6 in term 0.
I20260926 07:19:42.345793 1837 leader_election.cc:302] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b749b1e17224146ba2433db4a8e66ff, 6f3202a5abb54e0fa6a556d7334d33e6; no voters:
I20260926 07:19:42.346091 2233 raft_consensus.cc:2781] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:42.346203 2233 raft_consensus.cc:482] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:42.346262 2233 raft_consensus.cc:3037] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:42.347335 2233 raft_consensus.cc:504] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38335 } } peers { permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 } } peers { permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 } }
I20260926 07:19:42.347729 2233 leader_election.cc:288] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [CANDIDATE]: Term 1 election: Requested vote from peers 6b749b1e17224146ba2433db4a8e66ff (127.28.235.194:40231), 10038123eaf144369edb79b5c85de258 (127.28.235.195:35843)
I20260926 07:19:42.348120 2163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4875829129374bbfa2f686cfee7e0495" candidate_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10038123eaf144369edb79b5c85de258"
I20260926 07:19:42.348119 2032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4875829129374bbfa2f686cfee7e0495" candidate_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b749b1e17224146ba2433db4a8e66ff"
I20260926 07:19:42.348253 2032 raft_consensus.cc:3037] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:42.348253 2163 raft_consensus.cc:3037] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:42.349689 2163 raft_consensus.cc:2454] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f3202a5abb54e0fa6a556d7334d33e6 in term 1.
I20260926 07:19:42.349704 2032 raft_consensus.cc:2454] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f3202a5abb54e0fa6a556d7334d33e6 in term 1.
I20260926 07:19:42.350086 1834 leader_election.cc:302] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10038123eaf144369edb79b5c85de258, 6b749b1e17224146ba2433db4a8e66ff, 6f3202a5abb54e0fa6a556d7334d33e6; no voters:
I20260926 07:19:42.350276 2233 raft_consensus.cc:2781] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:42.350481 2233 raft_consensus.cc:686] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 1 LEADER]: Becoming Leader. State: Replica: 6f3202a5abb54e0fa6a556d7334d33e6, State: Running, Role: LEADER
I20260926 07:19:42.350816 2233 consensus_queue.cc:237] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38335 } } peers { permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 } } peers { permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 } }
I20260926 07:19:42.354388 1758 catalog_manager.cc:5495] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f3202a5abb54e0fa6a556d7334d33e6 (127.28.235.193). New cstate: current_term: 1 leader_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3202a5abb54e0fa6a556d7334d33e6" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:42.383466 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.381867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.28.235.193:38335)
0926 07:19:42.382016 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.382020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.382033 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:19:42.382116 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.382119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.382187 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.382261 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:42.383024 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:42.383027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:42.383029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:42.383039 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:42.383058 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:42.383066 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:42.383140 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:42.383250 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 07:19:42.383284 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:19:42.390228 2163 raft_consensus.cc:1261] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258 [term 1 FOLLOWER]: Refusing update from remote peer 6f3202a5abb54e0fa6a556d7334d33e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:42.390228 2032 raft_consensus.cc:1261] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff [term 1 FOLLOWER]: Refusing update from remote peer 6f3202a5abb54e0fa6a556d7334d33e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:42.391234 2237 consensus_queue.cc:1035] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b749b1e17224146ba2433db4a8e66ff" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:42.391526 2233 consensus_queue.cc:1035] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10038123eaf144369edb79b5c85de258" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:42.409044 1759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:42.409248 1759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:42.411996 1759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72481352d1ee48d69e50c57274cc3960: Sending DeleteTablet for 3 replicas of tablet 4875829129374bbfa2f686cfee7e0495
I20260926 07:19:42.412855 1881 tablet_service.cc:1514] Processing DeleteTablet for tablet 4875829129374bbfa2f686cfee7e0495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:42 UTC) from {username='slave'} at 127.0.0.1:39710
I20260926 07:19:42.412866 2012 tablet_service.cc:1514] Processing DeleteTablet for tablet 4875829129374bbfa2f686cfee7e0495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:42 UTC) from {username='slave'} at 127.0.0.1:35922
I20260926 07:19:42.413448 2250 tablet_replica.cc:330] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff: stopping tablet replica
I20260926 07:19:42.413699 2251 tablet_replica.cc:330] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6: stopping tablet replica
I20260926 07:19:42.414002 2251 raft_consensus.cc:2229] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:42.413955 2250 raft_consensus.cc:2229] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:42.414162 2143 tablet_service.cc:1514] Processing DeleteTablet for tablet 4875829129374bbfa2f686cfee7e0495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:42 UTC) from {username='slave'} at 127.0.0.1:58968
I20260926 07:19:42.414391 2251 raft_consensus.cc:2258] T 4875829129374bbfa2f686cfee7e0495 P 6f3202a5abb54e0fa6a556d7334d33e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:42.414626 2250 raft_consensus.cc:2258] T 4875829129374bbfa2f686cfee7e0495 P 6b749b1e17224146ba2433db4a8e66ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:42.414862 2252 tablet_replica.cc:330] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: stopping tablet replica
I20260926 07:19:42.415160 2252 raft_consensus.cc:2229] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:42.415269 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 1819
I20260926 07:19:42.415462 2252 raft_consensus.cc:2258] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:42.417523 2252 ts_tablet_manager.cc:1905] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:42.419823 2252 ts_tablet_manager.cc:1918] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:42.419947 2252 log.cc:1196] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/4875829129374bbfa2f686cfee7e0495
I20260926 07:19:42.420254 2252 ts_tablet_manager.cc:1939] T 4875829129374bbfa2f686cfee7e0495 P 10038123eaf144369edb79b5c85de258: Deleting consensus metadata
I20260926 07:19:42.421819 1743 catalog_manager.cc:4841] TS 10038123eaf144369edb79b5c85de258 (127.28.235.195:35843): tablet 4875829129374bbfa2f686cfee7e0495 (table test-table [id=c1c5f4d0b1d1468fa16d1d3b4b56ac6e]) successfully deleted
W20260926 07:19:42.422328 1744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:38335 (error 108)
I20260926 07:19:42.422497 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 1950
I20260926 07:19:42.426033 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.425631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38335 (local address 127.0.0.1:39742)
0926 07:19:42.425820 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:42.425916 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 07:19:42.426239 1744 catalog_manager.cc:4584] TS 6f3202a5abb54e0fa6a556d7334d33e6 (127.28.235.193:38335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4875829129374bbfa2f686cfee7e0495: Network error: Client connection negotiation failed: client connection to 127.28.235.193:38335: connect: Connection refused (error 111)
I20260926 07:19:42.428634 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 2081
I20260926 07:19:42.430151 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.428771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:40231 (local address 127.0.0.1:35932)
0926 07:19:42.430028 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:42.430077 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.194:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1171}
I20260926 07:19:42.434592 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 1728
2026-09-26T07:19:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1189 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 07:19:42.456974 29615 mini_oidc.cc:152] Starting JWKS server
I20260926 07:19:42.458668 29615 webserver.cc:460] Webserver started at https://127.0.0.1:35925/ using document root <none> and password file <none>
I20260926 07:19:42.458845 29615 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:19:42.459137 29615 webserver.cc:460] Webserver started at http://127.0.0.1:46711/ using document root <none> and password file <none>
2026-09-26T07:19:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:42Z Disabled control of system clock
I20260926 07:19:42.468657 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34953
--webserver_interface=127.28.235.254
--webserver_port=0
--builtin_ntp_servers=127.28.235.212:39603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:34953
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35925/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/testchainca.cert with env {}
W20260926 07:19:42.581662 2259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:42.581980 2259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:42.582031 2259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:42.586292 2259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:42.586374 2259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:42.586393 2259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:42.586419 2259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:42.586442 2259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:19:42.586462 2259 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:19:42.586479 2259 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35925/jwks.json
I20260926 07:19:42.592190 2259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:39603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:34953
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35925/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:42.593411 2259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:42.594529 2259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:42.603658 2265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:42.603670 2264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:42.603670 2267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:42.603896 2259 server_base.cc:1034] running on GCE node
I20260926 07:19:42.604605 2259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:42.605903 2259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:42.607120 2259 hybrid_clock.cc:648] HybridClock initialized: now 1790407182607104 us; error 27 us; skew 500 ppm
I20260926 07:19:42.609083 2259 webserver.cc:460] Webserver started at http://127.28.235.254:46075/ using document root <none> and password file <none>
I20260926 07:19:42.609683 2259 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:42.609776 2259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:42.610006 2259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:42.612021 2259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "230fdea94b704a859f47e3996e4ac774"
format_stamp: "Formatted at 2026-09-26 07:19:42 on dist-test-slave-7zlw"
server_key: "169fb24feaadf61491c2de32e376ed09"
server_key_iv: "afac32bb9fd295de1c05cf3ac014f36d"
server_key_version: "encryptionkey@0"
I20260926 07:19:42.612579 2259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "230fdea94b704a859f47e3996e4ac774"
format_stamp: "Formatted at 2026-09-26 07:19:42 on dist-test-slave-7zlw"
server_key: "169fb24feaadf61491c2de32e376ed09"
server_key_iv: "afac32bb9fd295de1c05cf3ac014f36d"
server_key_version: "encryptionkey@0"
I20260926 07:19:42.616238 2259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:19:42.618782 2273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:42.620051 2259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:19:42.620193 2259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "230fdea94b704a859f47e3996e4ac774"
format_stamp: "Formatted at 2026-09-26 07:19:42 on dist-test-slave-7zlw"
server_key: "169fb24feaadf61491c2de32e376ed09"
server_key_iv: "afac32bb9fd295de1c05cf3ac014f36d"
server_key_version: "encryptionkey@0"
I20260926 07:19:42.620302 2259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:42.632148 2259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:42.632850 2259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:42.633119 2259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:42.639930 2259 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:34953
I20260926 07:19:42.639932 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:34953 every 8 connection(s)
I20260926 07:19:42.640964 2259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:42.644315 2326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:42.644701 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 2259
I20260926 07:19:42.644845 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:42.645206 29615 external_mini_cluster.cc:1428] Setting key 3cb59865c087dc3ebbe8f418c95cc723
I20260926 07:19:42.651644 2326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774: Bootstrap starting.
I20260926 07:19:42.654565 2326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:42.655503 2326 log.cc:824] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:42.656327 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.646647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.28.235.254:34953)
0926 07:19:42.647388 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.647408 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.647464 (+ 56us) server_negotiation.cc:407] Connection header received
0926 07:19:42.648304 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.648331 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.648864 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.649277 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:42.649778 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:42.651237 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:42.652446 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:42.653191 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:42.653350 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:42.654190 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:42.654196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:42.654213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:42.654780 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:42.654859 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:42.654886 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:42.654978 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:42.655127 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 07:19:42.655411 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":101,"threads_started":1}
I20260926 07:19:42.657754 2326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774: No bootstrap required, opened a new log
I20260926 07:19:42.659199 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:19:42.661342 2326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230fdea94b704a859f47e3996e4ac774" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34953 } }
I20260926 07:19:42.661595 2326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:42.661672 2326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230fdea94b704a859f47e3996e4ac774, State: Initialized, Role: FOLLOWER
I20260926 07:19:42.662153 2326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230fdea94b704a859f47e3996e4ac774" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34953 } }
I20260926 07:19:42.662315 2326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:42.662401 2326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:42.662497 2326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:42.663600 2326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230fdea94b704a859f47e3996e4ac774" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34953 } }
I20260926 07:19:42.663951 2326 leader_election.cc:302] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230fdea94b704a859f47e3996e4ac774; no voters:
I20260926 07:19:42.664273 2326 leader_election.cc:288] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:42.664451 2331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:42.664697 2331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [term 1 LEADER]: Becoming Leader. State: Replica: 230fdea94b704a859f47e3996e4ac774, State: Running, Role: LEADER
I20260926 07:19:42.665005 2331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230fdea94b704a859f47e3996e4ac774" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34953 } }
I20260926 07:19:42.665596 2326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:42.666769 2332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "230fdea94b704a859f47e3996e4ac774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230fdea94b704a859f47e3996e4ac774" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34953 } } }
I20260926 07:19:42.666895 2332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:42.667182 2333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230fdea94b704a859f47e3996e4ac774. Latest consensus state: current_term: 1 leader_uuid: "230fdea94b704a859f47e3996e4ac774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230fdea94b704a859f47e3996e4ac774" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34953 } } }
I20260926 07:19:42.667269 2333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:42.667382 2340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:42.670439 2340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:42.676681 2340 catalog_manager.cc:1347] Generated new cluster ID: 4f55472ec36142b4a92b9f5bfd2077bc
I20260926 07:19:42.676774 2340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:42.691027 2340 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:42.691958 2340 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:42.697513 2340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 230fdea94b704a859f47e3996e4ac774: Generated new TSK 0
I20260926 07:19:42.698211 2340 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:19:42.731688 2356 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 07:19:42.731815 2356 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:19:42.732491 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.715006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59008 (local address 127.28.235.254:34953)
0926 07:19:42.715139 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.715144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.715300 (+ 156us) server_negotiation.cc:407] Connection header received
0926 07:19:42.715494 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.715497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.715555 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.715671 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:19:42.715898 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:42.716743 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:42.717659 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:42.718044 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:42.718148 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:42.718479 (+ 331us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:19:42.732017 (+ 13538us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:19:42.732309 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59008: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 07:19:42.732560 2329 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59008: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 07:19:42.733650 2355 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:19:42.714881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34953 (local address 127.0.0.1:59008)
0926 07:19:42.715212 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:42.715226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:42.715337 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:42.715687 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:42.715690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:42.715696 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:19:42.715797 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:42.715802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:42.716856 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:42.716858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:42.717549 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:42.717557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:42.718172 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:42.718175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:42.718242 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:42.718288 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:19:42.732261 (+ 13973us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:19:42.732461 (+ 200us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.235.254:34953: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":115,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.235.254:34953: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 07:19:42.734503 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 2259
2026-09-26T07:19:42Z chronyd exiting
I20260926 07:19:42.772599 29615 test_util.cc:175] -----------------------------------------------
I20260926 07:19:42.772687 29615 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407154183638-29615-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (317 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 07:19:42.773838 29615 mini_oidc.cc:152] Starting JWKS server
I20260926 07:19:42.774705 29615 webserver.cc:460] Webserver started at https://127.0.0.1:45371/ using document root <none> and password file <none>
I20260926 07:19:42.774785 29615 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:19:42.775089 29615 webserver.cc:460] Webserver started at http://127.0.0.1:35245/ using document root <none> and password file <none>
2026-09-26T07:19:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:42Z Disabled control of system clock
I20260926 07:19:42.784502 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:33693
--webserver_interface=127.28.235.254
--webserver_port=0
--builtin_ntp_servers=127.28.235.212:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:33693
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45371/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/testchainca.cert with env {}
W20260926 07:19:42.897425 2362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:42.897717 2362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:42.897759 2362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:42.901962 2362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:42.902025 2362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:42.902041 2362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:42.902060 2362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:42.902074 2362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:19:42.902091 2362 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:19:42.902104 2362 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45371/jwks.json
I20260926 07:19:42.907691 2362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:33693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:33693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45371/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:42.908735 2362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:42.909888 2362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:42.918442 2367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:42.918444 2370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:42.918547 2368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:42.918561 2362 server_base.cc:1034] running on GCE node
I20260926 07:19:42.919377 2362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:42.920539 2362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:42.921737 2362 hybrid_clock.cc:648] HybridClock initialized: now 1790407182921705 us; error 43 us; skew 500 ppm
I20260926 07:19:42.929903 2362 webserver.cc:460] Webserver started at http://127.28.235.254:45327/ using document root <none> and password file <none>
I20260926 07:19:42.930549 2362 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:42.930649 2362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:42.930894 2362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:42.933092 2362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "d8a1e2711a6242c99fab4ca851f855f5"
format_stamp: "Formatted at 2026-09-26 07:19:42 on dist-test-slave-7zlw"
server_key: "47cfc873b0248c50f54e78e6e9f69bbc"
server_key_iv: "b5dd776660bb33883ce9024ba28978a0"
server_key_version: "encryptionkey@0"
I20260926 07:19:42.933674 2362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "d8a1e2711a6242c99fab4ca851f855f5"
format_stamp: "Formatted at 2026-09-26 07:19:42 on dist-test-slave-7zlw"
server_key: "47cfc873b0248c50f54e78e6e9f69bbc"
server_key_iv: "b5dd776660bb33883ce9024ba28978a0"
server_key_version: "encryptionkey@0"
I20260926 07:19:42.937563 2362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:42.940217 2376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:42.941623 2362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:42.941768 2362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "d8a1e2711a6242c99fab4ca851f855f5"
format_stamp: "Formatted at 2026-09-26 07:19:42 on dist-test-slave-7zlw"
server_key: "47cfc873b0248c50f54e78e6e9f69bbc"
server_key_iv: "b5dd776660bb33883ce9024ba28978a0"
server_key_version: "encryptionkey@0"
I20260926 07:19:42.941887 2362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:42.963429 2362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:42.964262 2362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:42.964565 2362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:42.971913 2362 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:33693
I20260926 07:19:42.971915 2428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:33693 every 8 connection(s)
I20260926 07:19:42.972922 2362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:42.976294 2429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:42.980715 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 2362
I20260926 07:19:42.980840 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:42.981202 29615 external_mini_cluster.cc:1428] Setting key 6de5e2599a0ea67adf6452ccc3dcb196
I20260926 07:19:42.982234 2429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5: Bootstrap starting.
I20260926 07:19:42.984589 2429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:42.985502 2429 log.cc:824] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:42.987329 2429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5: No bootstrap required, opened a new log
I20260926 07:19:42.990389 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:42.982578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.28.235.254:33693)
0926 07:19:42.983201 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:42.983218 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:42.983254 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:19:42.983884 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:42.983901 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:42.984295 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:42.984587 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:42.985004 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:42.986116 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:42.987033 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:42.987554 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:42.987724 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:42.988591 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:42.988596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:42.988608 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:42.988984 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:42.989100 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:42.989122 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:42.989228 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:42.989344 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 07:19:42.989620 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":141,"threads_started":1}
I20260926 07:19:42.990976 2429 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a1e2711a6242c99fab4ca851f855f5" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33693 } }
I20260926 07:19:42.991226 2429 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:42.991302 2429 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a1e2711a6242c99fab4ca851f855f5, State: Initialized, Role: FOLLOWER
I20260926 07:19:42.991755 2429 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a1e2711a6242c99fab4ca851f855f5" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33693 } }
I20260926 07:19:42.991923 2429 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:42.992010 2429 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:42.992136 2429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:42.993254 2429 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a1e2711a6242c99fab4ca851f855f5" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33693 } }
I20260926 07:19:42.993433 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:19:42.993651 2429 leader_election.cc:302] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8a1e2711a6242c99fab4ca851f855f5; no voters:
I20260926 07:19:42.993953 2429 leader_election.cc:288] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:42.994100 2434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:42.994300 2434 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [term 1 LEADER]: Becoming Leader. State: Replica: d8a1e2711a6242c99fab4ca851f855f5, State: Running, Role: LEADER
I20260926 07:19:42.994683 2434 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a1e2711a6242c99fab4ca851f855f5" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33693 } }
I20260926 07:19:42.995292 2429 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:42.996434 2436 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8a1e2711a6242c99fab4ca851f855f5. Latest consensus state: current_term: 1 leader_uuid: "d8a1e2711a6242c99fab4ca851f855f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a1e2711a6242c99fab4ca851f855f5" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33693 } } }
I20260926 07:19:42.996572 2436 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:42.997711 2435 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8a1e2711a6242c99fab4ca851f855f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a1e2711a6242c99fab4ca851f855f5" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33693 } } }
I20260926 07:19:42.998104 2435 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5 [sys.catalog]: This master's current role is: LEADER
W20260926 07:19:43.000168 2449 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:19:43.000257 2449 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:19:43.000996 2443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:43.002058 2443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:43.008708 2443 catalog_manager.cc:1347] Generated new cluster ID: 7b60664423c84019b64d67843a7125fe
I20260926 07:19:43.008793 2443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:43.020764 2443 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:43.021767 2443 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:43.027297 2443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8a1e2711a6242c99fab4ca851f855f5: Generated new TSK 0
I20260926 07:19:43.028247 2443 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:19:43.065101 2459 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 07:19:43.065198 2459 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260926 07:19:43.066124 2458 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:19:43.041979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:33693 (local address 127.0.0.1:40240)
0926 07:19:43.042248 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:43.042260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:43.042350 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:43.042662 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:43.042664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:43.042670 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:19:43.042763 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:43.042767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.043814 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.043816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:43.044437 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:43.044443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.045074 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.045077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:43.045187 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:43.045219 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:19:43.065737 (+ 20518us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:19:43.065915 (+ 178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.235.254:33693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260926 07:19:43.066262 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:43.042079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.28.235.254:33693)
0926 07:19:43.042205 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:43.042209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:43.042341 (+ 132us) server_negotiation.cc:407] Connection header received
0926 07:19:43.042469 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:43.042472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:43.042535 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:43.042665 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:19:43.042873 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.043710 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.044554 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.044920 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.045051 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:43.045387 (+ 336us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:19:43.065497 (+ 20110us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:19:43.065829 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40240: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 07:19:43.066316 2432 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40240: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 07:19:43.066862 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 2362
2026-09-26T07:19:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 07:19:43.106567 29615 mini_oidc.cc:152] Starting JWKS server
I20260926 07:19:43.107513 29615 webserver.cc:460] Webserver started at https://127.0.0.1:43941/ using document root <none> and password file <none>
I20260926 07:19:43.107636 29615 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:19:43.107915 29615 webserver.cc:460] Webserver started at http://127.0.0.1:43547/ using document root <none> and password file <none>
2026-09-26T07:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:43Z Disabled control of system clock
I20260926 07:19:43.116740 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:45603
--webserver_interface=127.28.235.254
--webserver_port=0
--builtin_ntp_servers=127.28.235.212:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:45603
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43941/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:19:43.230835 2465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:43.231132 2465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:43.231172 2465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:43.235406 2465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:43.235486 2465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:43.235507 2465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:43.235526 2465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:43.235543 2465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:19:43.235562 2465 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:19:43.235576 2465 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43941/jwks.json
I20260926 07:19:43.240581 2465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:45603
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:45603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43941/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:43.241977 2465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:43.243383 2465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:43.252298 2470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:43.252285 2471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:43.252354 2473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:43.252620 2465 server_base.cc:1034] running on GCE node
I20260926 07:19:43.253463 2465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:43.254748 2465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:43.255955 2465 hybrid_clock.cc:648] HybridClock initialized: now 1790407183255930 us; error 34 us; skew 500 ppm
I20260926 07:19:43.258001 2465 webserver.cc:460] Webserver started at http://127.28.235.254:35173/ using document root <none> and password file <none>
I20260926 07:19:43.258610 2465 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:43.258673 2465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:43.258893 2465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:43.261090 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "a3b2342104c2444b9a350d4c19b2fac1"
format_stamp: "Formatted at 2026-09-26 07:19:43 on dist-test-slave-7zlw"
server_key: "1a3633436d7cf82975c3d4a41f9829db"
server_key_iv: "006cb66f0b9e9ca97f07e22f66a3e1bb"
server_key_version: "encryptionkey@0"
I20260926 07:19:43.261683 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "a3b2342104c2444b9a350d4c19b2fac1"
format_stamp: "Formatted at 2026-09-26 07:19:43 on dist-test-slave-7zlw"
server_key: "1a3633436d7cf82975c3d4a41f9829db"
server_key_iv: "006cb66f0b9e9ca97f07e22f66a3e1bb"
server_key_version: "encryptionkey@0"
I20260926 07:19:43.265527 2465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:19:43.268038 2479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:43.269613 2465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:43.269774 2465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "a3b2342104c2444b9a350d4c19b2fac1"
format_stamp: "Formatted at 2026-09-26 07:19:43 on dist-test-slave-7zlw"
server_key: "1a3633436d7cf82975c3d4a41f9829db"
server_key_iv: "006cb66f0b9e9ca97f07e22f66a3e1bb"
server_key_version: "encryptionkey@0"
I20260926 07:19:43.269898 2465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:43.286496 2465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:43.287195 2465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:43.287387 2465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:43.294796 2465 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:45603
I20260926 07:19:43.294816 2531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:45603 every 8 connection(s)
I20260926 07:19:43.295775 2465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:43.299008 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:43.302630 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 2465
I20260926 07:19:43.302790 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:43.303172 29615 external_mini_cluster.cc:1428] Setting key 301c19694756d2035fe9fe8e35b203f1
I20260926 07:19:43.306808 2532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1: Bootstrap starting.
I20260926 07:19:43.309725 2532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:43.310513 2532 log.cc:824] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:43.312868 2532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1: No bootstrap required, opened a new log
I20260926 07:19:43.313586 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:43.304665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.28.235.254:45603)
0926 07:19:43.305408 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:43.305430 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:43.305472 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:19:43.306223 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:43.306243 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:43.306674 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:43.306971 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:43.307416 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.308491 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.309832 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.310488 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.310649 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:43.311648 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:43.311653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:43.311666 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:43.312057 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:43.312115 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:43.312134 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:43.312226 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:43.312379 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 07:19:43.312696 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":154,"threads_started":1}
I20260926 07:19:43.316583 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:19:43.316663 2532 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2342104c2444b9a350d4c19b2fac1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45603 } }
I20260926 07:19:43.316922 2532 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:43.316999 2532 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b2342104c2444b9a350d4c19b2fac1, State: Initialized, Role: FOLLOWER
I20260926 07:19:43.317529 2532 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2342104c2444b9a350d4c19b2fac1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45603 } }
I20260926 07:19:43.317710 2532 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:43.317785 2532 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:43.317905 2532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:43.319017 2532 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2342104c2444b9a350d4c19b2fac1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45603 } }
I20260926 07:19:43.319314 2532 leader_election.cc:302] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3b2342104c2444b9a350d4c19b2fac1; no voters:
I20260926 07:19:43.319562 2532 leader_election.cc:288] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:43.319702 2537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:43.319980 2537 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [term 1 LEADER]: Becoming Leader. State: Replica: a3b2342104c2444b9a350d4c19b2fac1, State: Running, Role: LEADER
I20260926 07:19:43.320303 2537 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2342104c2444b9a350d4c19b2fac1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45603 } }
I20260926 07:19:43.320807 2532 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:43.322219 2538 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3b2342104c2444b9a350d4c19b2fac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2342104c2444b9a350d4c19b2fac1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45603 } } }
I20260926 07:19:43.322259 2539 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3b2342104c2444b9a350d4c19b2fac1. Latest consensus state: current_term: 1 leader_uuid: "a3b2342104c2444b9a350d4c19b2fac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2342104c2444b9a350d4c19b2fac1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45603 } } }
I20260926 07:19:43.322368 2538 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:43.322477 2539 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1 [sys.catalog]: This master's current role is: LEADER
W20260926 07:19:43.326139 2552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:19:43.326228 2552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:19:43.326298 2551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:43.327024 2551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:43.332917 2551 catalog_manager.cc:1347] Generated new cluster ID: 3839c808a2ad4f99926718bebe10e5d8
I20260926 07:19:43.332998 2551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:43.343426 2551 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:43.344738 2551 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:43.356079 2551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3b2342104c2444b9a350d4c19b2fac1: Generated new TSK 0
I20260926 07:19:43.356866 2551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:19:43.378243 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:43.372473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.28.235.254:45603)
0926 07:19:43.372620 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:43.372625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:43.372769 (+ 144us) server_negotiation.cc:407] Connection header received
0926 07:19:43.372929 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:43.372933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:43.372998 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:43.373157 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:19:43.373380 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.374263 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.375338 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.375701 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.375820 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:43.376206 (+ 386us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:19:43.377806 (+ 1600us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:19:43.378073 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45552: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 07:19:43.378325 2535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45552: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 07:19:43.379796 2561 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:19:43.372343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:45603 (local address 127.0.0.1:45552)
0926 07:19:43.372667 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:43.372680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:43.372789 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:43.373165 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:43.373167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:43.373175 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:19:43.373270 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:43.373275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.374392 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.374395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:43.375211 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:43.375221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.375859 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.375863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:43.375937 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:43.376012 (+ 75us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:19:43.377996 (+ 1984us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:19:43.379632 (+ 1636us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.235.254:45603: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":1446,"thread_start_us":129,"threads_started":1}
I20260926 07:19:43.380729 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 2465
E20260926 07:19:43.388957 2562 webserver.cc:533] Webserver: sslize failed:
E20260926 07:19:43.389073 2562 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T07:19:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (313 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T07:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:43Z Disabled control of system clock
I20260926 07:19:43.427992 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:46263
--webserver_interface=127.28.235.254
--webserver_port=0
--builtin_ntp_servers=127.28.235.212:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:46263
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/testchainca.cert with env {}
W20260926 07:19:43.538829 2566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:43.539220 2566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:43.539295 2566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:43.543462 2566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:43.543515 2566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:43.543530 2566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:43.543547 2566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:43.543562 2566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:19:43.543612 2566 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:19:43.543639 2566 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 07:19:43.549062 2566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:46263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:43.550191 2566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:43.551249 2566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:43.560043 2572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:43.560042 2574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:43.560040 2571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:43.560457 2566 server_base.cc:1034] running on GCE node
I20260926 07:19:43.561106 2566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:43.562256 2566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:43.563408 2566 hybrid_clock.cc:648] HybridClock initialized: now 1790407183563379 us; error 45 us; skew 500 ppm
I20260926 07:19:43.565364 2566 webserver.cc:460] Webserver started at http://127.28.235.254:46247/ using document root <none> and password file <none>
I20260926 07:19:43.565953 2566 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:43.566015 2566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:43.566232 2566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:43.568288 2566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "a1dfb534db2740b5b080d72455aa0ff6"
format_stamp: "Formatted at 2026-09-26 07:19:43 on dist-test-slave-7zlw"
server_key: "f95e1c0f57f2b36556247ce7f9fad841"
server_key_iv: "3ffb3a3118a6241a856a28894f6083c6"
server_key_version: "encryptionkey@0"
I20260926 07:19:43.568864 2566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "a1dfb534db2740b5b080d72455aa0ff6"
format_stamp: "Formatted at 2026-09-26 07:19:43 on dist-test-slave-7zlw"
server_key: "f95e1c0f57f2b36556247ce7f9fad841"
server_key_iv: "3ffb3a3118a6241a856a28894f6083c6"
server_key_version: "encryptionkey@0"
I20260926 07:19:43.572716 2566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 07:19:43.575307 2580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:43.576727 2566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:19:43.576854 2566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "a1dfb534db2740b5b080d72455aa0ff6"
format_stamp: "Formatted at 2026-09-26 07:19:43 on dist-test-slave-7zlw"
server_key: "f95e1c0f57f2b36556247ce7f9fad841"
server_key_iv: "3ffb3a3118a6241a856a28894f6083c6"
server_key_version: "encryptionkey@0"
I20260926 07:19:43.576967 2566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:43.588119 2566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:43.588790 2566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:43.588987 2566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:43.596158 2566 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:46263
I20260926 07:19:43.596166 2632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:46263 every 8 connection(s)
I20260926 07:19:43.597136 2566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:43.600006 2633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:43.603856 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 2566
I20260926 07:19:43.604014 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:43.604326 29615 external_mini_cluster.cc:1428] Setting key d37436257dd8994f7c0e56cdd3d0f26b
I20260926 07:19:43.606101 2633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6: Bootstrap starting.
I20260926 07:19:43.608496 2633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:43.609408 2633 log.cc:824] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:43.611599 2633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6: No bootstrap required, opened a new log
I20260926 07:19:43.613441 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:43.605660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.28.235.254:46263)
0926 07:19:43.606332 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:43.606348 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:43.606383 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:19:43.607008 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:43.607024 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:43.607416 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:43.607697 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:43.608034 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.609097 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.610211 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.610665 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.610778 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:43.611682 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:43.611692 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:43.611709 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:19:43.612102 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:43.612160 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:19:43.612180 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:43.612257 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:43.612418 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 07:19:43.612708 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260926 07:19:43.615374 2633 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dfb534db2740b5b080d72455aa0ff6" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46263 } }
I20260926 07:19:43.615660 2633 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:43.615741 2633 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1dfb534db2740b5b080d72455aa0ff6, State: Initialized, Role: FOLLOWER
I20260926 07:19:43.616216 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:19:43.616228 2633 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dfb534db2740b5b080d72455aa0ff6" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46263 } }
I20260926 07:19:43.616400 2633 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:43.616467 2633 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:43.616562 2633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:43.617662 2633 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dfb534db2740b5b080d72455aa0ff6" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46263 } }
I20260926 07:19:43.618012 2633 leader_election.cc:302] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1dfb534db2740b5b080d72455aa0ff6; no voters:
I20260926 07:19:43.618327 2633 leader_election.cc:288] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:43.618399 2638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:43.618698 2638 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [term 1 LEADER]: Becoming Leader. State: Replica: a1dfb534db2740b5b080d72455aa0ff6, State: Running, Role: LEADER
I20260926 07:19:43.618990 2638 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dfb534db2740b5b080d72455aa0ff6" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46263 } }
I20260926 07:19:43.619702 2633 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:43.620698 2639 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1dfb534db2740b5b080d72455aa0ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dfb534db2740b5b080d72455aa0ff6" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46263 } } }
I20260926 07:19:43.620823 2639 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:43.621129 2640 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1dfb534db2740b5b080d72455aa0ff6. Latest consensus state: current_term: 1 leader_uuid: "a1dfb534db2740b5b080d72455aa0ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dfb534db2740b5b080d72455aa0ff6" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46263 } } }
I20260926 07:19:43.621225 2640 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:43.621258 2647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:43.624396 2647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:43.629362 2647 catalog_manager.cc:1347] Generated new cluster ID: 5af2d80a0daa4fff97ce848fdc0133f4
I20260926 07:19:43.629431 2647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:43.641075 2647 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:43.642292 2647 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:43.649190 2647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1dfb534db2740b5b080d72455aa0ff6: Generated new TSK 0
I20260926 07:19:43.649974 2647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:19:43.732146 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:43.661725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.28.235.254:46263)
0926 07:19:43.661880 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:43.661887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:43.661903 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:19:43.662106 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:43.662108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:43.662158 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:43.662271 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:19:43.662520 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.663099 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.663963 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.664196 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.664307 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:43.664621 (+ 314us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:19:43.731649 (+ 67028us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:19:43.731985 (+ 336us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38732: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 07:19:43.732245 2636 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38732: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 07:19:43.732332 2662 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:19:43.661557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:46263 (local address 127.0.0.1:38732)
0926 07:19:43.661831 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:43.661846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:43.661960 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:43.662283 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:43.662286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:43.662295 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:19:43.662411 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:43.662417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.663217 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.663221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:43.663842 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:43.663849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:43.664323 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:43.664327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:43.664413 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:43.664436 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:19:43.731896 (+ 67460us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:19:43.732135 (+ 239us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.235.254:46263: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":112,"threads_started":1}
I20260926 07:19:43.733062 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 2566
2026-09-26T07:19:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (332 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (124 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:44 dist-test-slave-7zlw krb5kdc[2674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:44 dist-test-slave-7zlw krb5kdc[2674](info): set up 2 sockets
Sep 26 07:19:44 dist-test-slave-7zlw krb5kdc[2674](info): commencing operation
krb5kdc: starting...
W20260926 07:19:45.038080 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:45 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:45.168120 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:41315
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:41315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:45.281731 2687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:45.282076 2687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:45.282191 2687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:45.286350 2687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:45.286445 2687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:45.286484 2687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:45.286522 2687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:45.286551 2687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:45.292111 2687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:41315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:41315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:45.293329 2687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:45.294428 2687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:45.303288 2695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:45.303320 2693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:45.303328 2692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:45.303689 2687 server_base.cc:1034] running on GCE node
I20260926 07:19:45.304487 2687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:45.305864 2687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:45.307120 2687 hybrid_clock.cc:648] HybridClock initialized: now 1790407185307087 us; error 50 us; skew 500 ppm
Sep 26 07:19:45 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:45.310339 2687 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:45.311378 2687 webserver.cc:460] Webserver started at http://127.28.235.254:38647/ using document root <none> and password file <none>
I20260926 07:19:45.312170 2687 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:45.312282 2687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:45.312568 2687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:45.315062 2687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65"
format_stamp: "Formatted at 2026-09-26 07:19:45 on dist-test-slave-7zlw"
server_key: "ce9a593f6e27ae8ed61b5d7d6e24945d"
server_key_iv: "fa690913b46af4b3032e6925bc98705f"
server_key_version: "encryptionkey@0"
I20260926 07:19:45.315718 2687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65"
format_stamp: "Formatted at 2026-09-26 07:19:45 on dist-test-slave-7zlw"
server_key: "ce9a593f6e27ae8ed61b5d7d6e24945d"
server_key_iv: "fa690913b46af4b3032e6925bc98705f"
server_key_version: "encryptionkey@0"
I20260926 07:19:45.320168 2687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 07:19:45.323257 2702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:45.324882 2687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:45.325081 2687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65"
format_stamp: "Formatted at 2026-09-26 07:19:45 on dist-test-slave-7zlw"
server_key: "ce9a593f6e27ae8ed61b5d7d6e24945d"
server_key_iv: "fa690913b46af4b3032e6925bc98705f"
server_key_version: "encryptionkey@0"
I20260926 07:19:45.325220 2687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:45.339115 2687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:45.342492 2687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:45.342730 2687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:45.351096 2687 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:41315
I20260926 07:19:45.351133 2754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:41315 every 8 connection(s)
I20260926 07:19:45.352128 2687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:45.355113 2755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:45.355213 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 2687
I20260926 07:19:45.355351 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:45.355706 29615 external_mini_cluster.cc:1428] Setting key e4b07315440d84a4fc317757440ebe77
I20260926 07:19:45.362142 2755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65: Bootstrap starting.
I20260926 07:19:45.364861 2755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65: Neither blocks nor log segments found. Creating new log.
Sep 26 07:19:45 dist-test-slave-7zlw krb5kdc[2674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:45.365804 2755 log.cc:824] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:45.368341 2755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65: No bootstrap required, opened a new log
I20260926 07:19:45.371641 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:45.357169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.28.235.254:41315)
0926 07:19:45.357600 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:45.357617 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:45.357653 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:19:45.358413 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:45.358434 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:45.358780 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:45.359086 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:45.360039 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:45.361153 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:45.362322 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:45.362989 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:45.363150 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:45.366027 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:45.366034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:45.366051 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:45.366083 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:45.368112 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:45.368641 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:45.368646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:45.368653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:45.368794 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:45.369199 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:45.369202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:45.369204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:45.369727 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:45.369922 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:45.370255 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 07:19:45.370590 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":132,"threads_started":1}
I20260926 07:19:45.373135 2755 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41315 } }
I20260926 07:19:45.373396 2755 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:45.373459 2755 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb7dc62ae36a43ebaeb7dc47d2efcd65, State: Initialized, Role: FOLLOWER
I20260926 07:19:45.373960 2755 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41315 } }
I20260926 07:19:45.374137 2755 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:45.374246 2755 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:45.374347 2755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:45.375288 2755 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41315 } }
I20260926 07:19:45.375629 2755 leader_election.cc:302] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb7dc62ae36a43ebaeb7dc47d2efcd65; no voters:
I20260926 07:19:45.375952 2755 leader_election.cc:288] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:45.376117 2760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:45.376349 2760 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [term 1 LEADER]: Becoming Leader. State: Replica: cb7dc62ae36a43ebaeb7dc47d2efcd65, State: Running, Role: LEADER
I20260926 07:19:45.376843 2760 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41315 } }
I20260926 07:19:45.377345 2755 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:45.378736 2762 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb7dc62ae36a43ebaeb7dc47d2efcd65. Latest consensus state: current_term: 1 leader_uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41315 } } }
I20260926 07:19:45.378858 2762 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:45.378818 2761 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7dc62ae36a43ebaeb7dc47d2efcd65" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41315 } } }
I20260926 07:19:45.378918 2761 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:45.379436 2770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:45.382208 2770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:45.387687 2770 catalog_manager.cc:1347] Generated new cluster ID: 596272d598414d82a8e92a1f6c3e3ea4
I20260926 07:19:45.387753 2770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:45.402009 2770 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:45.403326 2770 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:45.413918 2770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb7dc62ae36a43ebaeb7dc47d2efcd65: Generated new TSK 0
I20260926 07:19:45.414790 2770 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:19:45.457604 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:41315
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:45.571753 2781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:45.572115 2781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:45.572199 2781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:45.576435 2781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:45.576567 2781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:45.576670 2781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:45.582517 2781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:41315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:45.583665 2781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:45.584733 2781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:45.594599 2787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:45.594796 2781 server_base.cc:1034] running on GCE node
W20260926 07:19:45.594599 2786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:45.594635 2789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:45.595492 2781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:45.596127 2781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:45.597301 2781 hybrid_clock.cc:648] HybridClock initialized: now 1790407185597262 us; error 41 us; skew 500 ppm
Sep 26 07:19:45 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:45.600306 2781 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:19:45.601328 2781 webserver.cc:460] Webserver started at http://127.28.235.193:36985/ using document root <none> and password file <none>
I20260926 07:19:45.601929 2781 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:45.601994 2781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:45.602216 2781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:45.604393 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "7e472a329d10473db304c6608d09e244"
format_stamp: "Formatted at 2026-09-26 07:19:45 on dist-test-slave-7zlw"
server_key: "294034053903a4c4131d411c9839d2fd"
server_key_iv: "d636df6a5f87e9f54340a47ea4c13d69"
server_key_version: "encryptionkey@0"
I20260926 07:19:45.605001 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "7e472a329d10473db304c6608d09e244"
format_stamp: "Formatted at 2026-09-26 07:19:45 on dist-test-slave-7zlw"
server_key: "294034053903a4c4131d411c9839d2fd"
server_key_iv: "d636df6a5f87e9f54340a47ea4c13d69"
server_key_version: "encryptionkey@0"
I20260926 07:19:45.609189 2781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:19:45.612221 2796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:45.613934 2781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:45.614094 2781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "7e472a329d10473db304c6608d09e244"
format_stamp: "Formatted at 2026-09-26 07:19:45 on dist-test-slave-7zlw"
server_key: "294034053903a4c4131d411c9839d2fd"
server_key_iv: "d636df6a5f87e9f54340a47ea4c13d69"
server_key_version: "encryptionkey@0"
I20260926 07:19:45.614243 2781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:45.627024 2781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:45.630523 2781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:45.630734 2781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:45.631330 2781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:45.632153 2781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:45.632225 2781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:45.632295 2781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:45.632350 2781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:45.641875 2781 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:36335
I20260926 07:19:45.641896 2909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:36335 every 8 connection(s)
I20260926 07:19:45.642777 2781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:45.644461 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 2781
I20260926 07:19:45.644593 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:45.644909 29615 external_mini_cluster.cc:1428] Setting key 036a1e2f13298eee39376b36b213f8d7
Sep 26 07:19:45 dist-test-slave-7zlw krb5kdc[2674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:19:45.659492 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:45.644969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:45379 (local address 127.28.235.254:41315)
0926 07:19:45.645148 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:45.645153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:45.646044 (+ 891us) server_negotiation.cc:407] Connection header received
0926 07:19:45.647429 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:45.647435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:45.647506 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:45.647599 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:45.649399 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:45.649946 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:45.651800 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:45.652051 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:45.652136 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:45.655220 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:45.655224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:45.655226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:45.655252 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:45.656668 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:45.657416 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:45.657420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:45.657421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:45.657510 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:45.657986 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:45.657989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:45.657990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:45.658202 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:45.658300 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:45.659102 (+ 802us) server_negotiation.cc:299] Negotiation successful
0926 07:19:45.659262 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:19:45.660259 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:45.645410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:41315 (local address 127.28.235.193:45379)
0926 07:19:45.645873 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:45.645925 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:45.647225 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:45.647841 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:45.647852 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:45.648475 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:45.649223 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:45.649263 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:45.650083 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:45.650090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:45.651655 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:45.651670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:45.652217 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:45.652223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:45.652432 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:45.653276 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:45.653302 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:45.655023 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:45.656777 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:45.656781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:45.656795 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:45.657297 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:45.657613 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:45.657617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:45.657619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:45.657794 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:45.658329 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:45.658338 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:45.658715 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 07:19:45.658929 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 07:19:45.659293 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":136,"threads_started":1}
I20260926 07:19:45.661655 2910 heartbeater.cc:347] Connected to a master server at 127.28.235.254:41315
I20260926 07:19:45.661896 2910 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:45.662400 2910 heartbeater.cc:511] Master 127.28.235.254:41315 requested a full tablet report, sending...
I20260926 07:19:45.663901 2719 ts_manager.cc:194] Registered new tserver with Master: 7e472a329d10473db304c6608d09e244 (127.28.235.193:36335)
I20260926 07:19:45.665160 2719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:45379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:19:45.676584 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:41315
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:45.788823 2915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:45.789194 2915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:45.789280 2915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:45.793672 2915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:45.793756 2915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:45.793843 2915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:45.799181 2915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:41315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:45.800269 2915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:45.801425 2915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:45.811848 2921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:45.811872 2923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:45.811872 2920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:45.812327 2915 server_base.cc:1034] running on GCE node
I20260926 07:19:45.812769 2915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:45.813534 2915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:45.814718 2915 hybrid_clock.cc:648] HybridClock initialized: now 1790407185814699 us; error 35 us; skew 500 ppm
Sep 26 07:19:45 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:45.817812 2915 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:19:45.818672 2915 webserver.cc:460] Webserver started at http://127.28.235.194:46075/ using document root <none> and password file <none>
I20260926 07:19:45.819196 2915 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:45.819252 2915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:45.819414 2915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:45.821473 2915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "52d569dacb984d969d465d8a5f5063d0"
format_stamp: "Formatted at 2026-09-26 07:19:45 on dist-test-slave-7zlw"
server_key: "ec78d7ac020040b19ec07c319e9b6c4c"
server_key_iv: "98e06d36e8e1399a9aaf6d8cc4582f9d"
server_key_version: "encryptionkey@0"
I20260926 07:19:45.821966 2915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "52d569dacb984d969d465d8a5f5063d0"
format_stamp: "Formatted at 2026-09-26 07:19:45 on dist-test-slave-7zlw"
server_key: "ec78d7ac020040b19ec07c319e9b6c4c"
server_key_iv: "98e06d36e8e1399a9aaf6d8cc4582f9d"
server_key_version: "encryptionkey@0"
I20260926 07:19:45.826054 2915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:19:45.828635 2930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:45.830200 2915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:45.830335 2915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "52d569dacb984d969d465d8a5f5063d0"
format_stamp: "Formatted at 2026-09-26 07:19:45 on dist-test-slave-7zlw"
server_key: "ec78d7ac020040b19ec07c319e9b6c4c"
server_key_iv: "98e06d36e8e1399a9aaf6d8cc4582f9d"
server_key_version: "encryptionkey@0"
I20260926 07:19:45.830454 2915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:45.861466 2915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:45.865036 2915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:45.865262 2915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:45.865903 2915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:45.866776 2915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:45.866850 2915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:45.866923 2915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:45.866976 2915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:45.876058 2915 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:35079
I20260926 07:19:45.876080 3043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:35079 every 8 connection(s)
I20260926 07:19:45.876992 2915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:45.883769 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 2915
I20260926 07:19:45.883921 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:45.884244 29615 external_mini_cluster.cc:1428] Setting key c652fd86282a6a9bb4ea561bb4b14666
Sep 26 07:19:45 dist-test-slave-7zlw krb5kdc[2674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407185, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:45.892850 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:45.879235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:60301 (local address 127.28.235.254:41315)
0926 07:19:45.879406 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:45.879410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:45.880320 (+ 910us) server_negotiation.cc:407] Connection header received
0926 07:19:45.881747 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:45.881750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:45.881797 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:45.881873 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:45.883229 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:45.883761 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:45.885293 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:45.885645 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:45.885757 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:45.888651 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:45.888658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:45.888664 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:45.888698 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:45.890479 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:45.891067 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:45.891071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:45.891072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:45.891171 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:45.891537 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:45.891544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:45.891548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:45.891824 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:45.891958 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:45.892485 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 07:19:45.892635 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:19:45.893694 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:45.879687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:41315 (local address 127.28.235.194:60301)
0926 07:19:45.880153 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:45.880200 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:45.881525 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:45.882147 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:45.882155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:45.882534 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:45.883069 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:45.883084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:45.883907 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:45.883913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:45.885175 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:45.885185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:45.885812 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:45.885816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:45.886023 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:45.886718 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:45.886744 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:45.888476 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:45.890604 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:45.890609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:45.890622 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:45.890957 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:45.891275 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:45.891278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:45.891279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:45.891420 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:45.891938 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:45.891944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:45.892219 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 07:19:45.892384 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 07:19:45.892657 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:19:45.894977 3044 heartbeater.cc:347] Connected to a master server at 127.28.235.254:41315
I20260926 07:19:45.895241 3044 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:45.895735 3044 heartbeater.cc:511] Master 127.28.235.254:41315 requested a full tablet report, sending...
I20260926 07:19:45.896816 2719 ts_manager.cc:194] Registered new tserver with Master: 52d569dacb984d969d465d8a5f5063d0 (127.28.235.194:35079)
I20260926 07:19:45.897475 2719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:60301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:19:45.917840 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:41315
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:46.035833 3049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:46.036188 3049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:46.036262 3049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:46.041365 3049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:46.041429 3049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:46.041517 3049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:46.047124 3049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:41315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:46.048261 3049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:46.049391 3049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:46.059321 3054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:46.059312 3055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:46.059321 3057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:46.059451 3049 server_base.cc:1034] running on GCE node
I20260926 07:19:46.060077 3049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:46.060742 3049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:46.061923 3049 hybrid_clock.cc:648] HybridClock initialized: now 1790407186061897 us; error 40 us; skew 500 ppm
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:46.064954 3049 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:19:46.066054 3049 webserver.cc:460] Webserver started at http://127.28.235.195:45025/ using document root <none> and password file <none>
I20260926 07:19:46.066668 3049 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:46.066738 3049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:46.066974 3049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:46.069247 3049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "42d01e8aa56c436e9bddd59163eddf67"
format_stamp: "Formatted at 2026-09-26 07:19:46 on dist-test-slave-7zlw"
server_key: "90213d948b2581bf6188c50c8c863e51"
server_key_iv: "8996e22b5d8c6adc136b78b02525e30f"
server_key_version: "encryptionkey@0"
I20260926 07:19:46.069893 3049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "42d01e8aa56c436e9bddd59163eddf67"
format_stamp: "Formatted at 2026-09-26 07:19:46 on dist-test-slave-7zlw"
server_key: "90213d948b2581bf6188c50c8c863e51"
server_key_iv: "8996e22b5d8c6adc136b78b02525e30f"
server_key_version: "encryptionkey@0"
I20260926 07:19:46.074234 3049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:19:46.076995 3064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:46.078770 3049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:19:46.078929 3049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "42d01e8aa56c436e9bddd59163eddf67"
format_stamp: "Formatted at 2026-09-26 07:19:46 on dist-test-slave-7zlw"
server_key: "90213d948b2581bf6188c50c8c863e51"
server_key_iv: "8996e22b5d8c6adc136b78b02525e30f"
server_key_version: "encryptionkey@0"
I20260926 07:19:46.079104 3049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:46.096329 3049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:46.099774 3049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:46.099992 3049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:46.100618 3049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:46.101670 3049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:46.101740 3049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:46.101814 3049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:46.101866 3049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:46.111182 3049 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:33423
I20260926 07:19:46.111202 3177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:33423 every 8 connection(s)
I20260926 07:19:46.112185 3049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:46.115042 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 3049
I20260926 07:19:46.115165 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:46.115515 29615 external_mini_cluster.cc:1428] Setting key ba0b17bea10fab954ba2ef26a6ac147b
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:46.127028 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:46.114145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:57227 (local address 127.28.235.254:41315)
0926 07:19:46.114307 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:46.114311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:46.115341 (+ 1030us) server_negotiation.cc:407] Connection header received
0926 07:19:46.117032 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:46.117036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:46.117102 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:46.117190 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:46.118463 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:46.119014 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:46.120292 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:46.120519 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:46.120619 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:46.123200 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:46.123203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:46.123205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:46.123238 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:46.125190 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:46.125693 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:46.125696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:46.125697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:46.125770 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:46.126078 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:46.126081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:46.126082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:46.126259 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:46.126337 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:46.126810 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 07:19:46.126910 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:19:46.127864 3180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:46.114524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:41315 (local address 127.28.235.195:57227)
0926 07:19:46.115152 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:46.115216 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:46.116763 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:46.117384 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:46.117391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:46.117795 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:46.118307 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:46.118320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:46.119148 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:46.119151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:46.120141 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:46.120150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:46.120661 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:46.120663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:46.120805 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:46.121480 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:46.121497 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:46.123036 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:46.125318 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:46.125322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:46.125335 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:46.125582 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:46.125849 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:46.125851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:46.125852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:46.125977 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:46.126349 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:46.126353 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:46.126579 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 07:19:46.126759 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 07:19:46.127008 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":113,"threads_started":1}
I20260926 07:19:46.129156 3178 heartbeater.cc:347] Connected to a master server at 127.28.235.254:41315
I20260926 07:19:46.129417 3178 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:46.130064 3178 heartbeater.cc:511] Master 127.28.235.254:41315 requested a full tablet report, sending...
I20260926 07:19:46.131127 2718 ts_manager.cc:194] Registered new tserver with Master: 42d01e8aa56c436e9bddd59163eddf67 (127.28.235.195:33423)
I20260926 07:19:46.131727 2718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:57227
I20260926 07:19:46.142365 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/test-admin.keytab.
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:46.159494 29615 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 07:19:46.162519 3185 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:19:46.160973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.28.235.193:36335)
0926 07:19:46.161309 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:46.161319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:46.161361 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:19:46.161417 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:46.161422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:46.161611 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:46.161743 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:46.162328 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38360: BlockingRecv error: recv got EOF from 127.0.0.1:38360 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":167,"threads_started":1}
W20260926 07:19:46.162700 29615 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:46.163856 29615 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:46.164885 29615 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:19:46.175911 3185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:46.166058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.28.235.193:36335)
0926 07:19:46.166234 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:46.166238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:46.166254 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:19:46.166309 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:46.166311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:46.166351 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:46.166445 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:46.167043 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:46.167808 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:46.168679 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:46.169151 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:46.169270 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:46.171851 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:46.171870 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:46.171889 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:46.171924 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:46.173837 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:46.174413 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:46.174420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:46.174421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:46.174520 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:46.174883 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:46.174887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:46.174888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:46.175193 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:46.175371 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:46.175622 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 07:19:46.175772 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260926 07:19:46.180418 3185 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:19:46.179298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.28.235.193:36335)
0926 07:19:46.179426 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:46.179431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:46.179589 (+ 158us) server_negotiation.cc:407] Connection header received
0926 07:19:46.179748 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:46.179751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:46.179796 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:46.179881 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:46.180292 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38376: BlockingRecv error: recv got EOF from 127.0.0.1:38376 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 07:19:46.180613 29615 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:46.181800 29615 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:46.182894 29615 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:19:46.192727 3185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:46.183802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.28.235.193:36335)
0926 07:19:46.183970 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:46.183975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:46.184134 (+ 159us) server_negotiation.cc:407] Connection header received
0926 07:19:46.184186 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:46.184188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:46.184230 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:46.184314 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:46.184809 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:46.185533 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:46.186463 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:46.186686 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:46.186777 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:46.189337 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:46.189340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:46.189342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:46.189366 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:46.190934 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:46.191445 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:46.191448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:46.191450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:46.191526 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:46.191816 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:46.191822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:46.191824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:46.192006 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:46.192168 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:46.192425 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 07:19:46.192556 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 07:19:46.195927 3185 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:19:46.194643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38388 (local address 127.28.235.193:36335)
0926 07:19:46.194794 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:46.194798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:46.194933 (+ 135us) server_negotiation.cc:407] Connection header received
0926 07:19:46.195158 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:46.195162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:46.195207 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:46.195299 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:46.195820 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38388: BlockingRecv error: recv got EOF from 127.0.0.1:38388 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 07:19:46.196084 29615 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407154183638-29615-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:46.197237 29615 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:46.198163 29615 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[2674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:19:46.207435 3185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:46.199078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.28.235.193:36335)
0926 07:19:46.199212 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:46.199216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:46.199230 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:19:46.199330 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:46.199332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:46.199367 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:46.199448 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:46.199939 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:46.200442 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:46.201367 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:46.201589 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:46.201696 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:46.204203 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:46.204211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:46.204213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:46.204235 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:46.205782 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:46.206272 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:46.206277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:46.206278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:46.206358 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:46.206642 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:46.206646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:46.206648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:46.206812 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:46.207036 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:46.207162 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 07:19:46.207264 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:19:46.208513 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 2781
I20260926 07:19:46.214488 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 2915
I20260926 07:19:46.220065 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 3049
I20260926 07:19:46.225730 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 2687
2026-09-26T07:19:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2247 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[3198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[3198](info): set up 2 sockets
Sep 26 07:19:46 dist-test-slave-7zlw krb5kdc[3198](info): commencing operation
krb5kdc: starting...
W20260926 07:19:47.325729 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:47 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.254.keytab.
Entry for principal oryx/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.254.keytab.
I20260926 07:19:47.453522 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34501
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.254.keytab
--principal=oryx/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:34501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:47.571285 3211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:47.571597 3211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:47.571645 3211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:47.575803 3211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:47.575886 3211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:47.575917 3211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:47.575937 3211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:47.575956 3211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:47.581610 3211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:34501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.254.keytab
--principal=oryx/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:47.582764 3211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:47.583925 3211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:47.592829 3216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:47.592847 3217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:47.592846 3219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:47.593201 3211 server_base.cc:1034] running on GCE node
I20260926 07:19:47.593760 3211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:47.594905 3211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:47.596110 3211 hybrid_clock.cc:648] HybridClock initialized: now 1790407187596083 us; error 49 us; skew 500 ppm
Sep 26 07:19:47 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407187, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:47.599004 3211 init.cc:377] Logged in from keytab as oryx/127.28.235.254@KRBTEST.COM (short username oryx)
I20260926 07:19:47.599988 3211 webserver.cc:460] Webserver started at http://127.28.235.254:42517/ using document root <none> and password file <none>
I20260926 07:19:47.600554 3211 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:47.600646 3211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:47.600873 3211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:47.602977 3211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "3335d929499c4a7dae82c7ae2aa00496"
format_stamp: "Formatted at 2026-09-26 07:19:47 on dist-test-slave-7zlw"
server_key: "09c9e5c67e86bbae0673c0311e8e64c2"
server_key_iv: "b2c08d6da2bf9d04987ab69136c942ab"
server_key_version: "encryptionkey@0"
I20260926 07:19:47.603571 3211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "3335d929499c4a7dae82c7ae2aa00496"
format_stamp: "Formatted at 2026-09-26 07:19:47 on dist-test-slave-7zlw"
server_key: "09c9e5c67e86bbae0673c0311e8e64c2"
server_key_iv: "b2c08d6da2bf9d04987ab69136c942ab"
server_key_version: "encryptionkey@0"
I20260926 07:19:47.607374 3211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260926 07:19:47.610116 3226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:47.611449 3211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:19:47.611609 3211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "3335d929499c4a7dae82c7ae2aa00496"
format_stamp: "Formatted at 2026-09-26 07:19:47 on dist-test-slave-7zlw"
server_key: "09c9e5c67e86bbae0673c0311e8e64c2"
server_key_iv: "b2c08d6da2bf9d04987ab69136c942ab"
server_key_version: "encryptionkey@0"
I20260926 07:19:47.611726 3211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:47.626999 3211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:47.631063 3211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:47.631281 3211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:47.639653 3211 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:34501
I20260926 07:19:47.640028 3278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:34501 every 8 connection(s)
I20260926 07:19:47.640895 3211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:47.644336 3279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:47.650072 3279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496: Bootstrap starting.
I20260926 07:19:47.650831 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 3211
I20260926 07:19:47.651046 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:47.651392 29615 external_mini_cluster.cc:1428] Setting key 23e3cfec54ac91842c59ea1b34a44ee8
I20260926 07:19:47.652755 3279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:47.653561 3279 log.cc:824] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:47.655717 3279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496: No bootstrap required, opened a new log
I20260926 07:19:47.660070 3279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } }
I20260926 07:19:47.660367 3279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:47.660458 3279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3335d929499c4a7dae82c7ae2aa00496, State: Initialized, Role: FOLLOWER
Sep 26 07:19:47 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:47.660959 3279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } }
I20260926 07:19:47.661175 3279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:47.661266 3279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:47.661366 3279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:47.662287 3279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } }
I20260926 07:19:47.662647 3279 leader_election.cc:302] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3335d929499c4a7dae82c7ae2aa00496; no voters:
I20260926 07:19:47.663034 3279 leader_election.cc:288] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:47.663214 3284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:47.663461 3284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 1 LEADER]: Becoming Leader. State: Replica: 3335d929499c4a7dae82c7ae2aa00496, State: Running, Role: LEADER
I20260926 07:19:47.663827 3284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } }
I20260926 07:19:47.664467 3279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:47.665799 3286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3335d929499c4a7dae82c7ae2aa00496. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } }
I20260926 07:19:47.665925 3286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:47.666308 3293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:47.666296 3285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } }
I20260926 07:19:47.666426 3285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:47.668005 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:47.652597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.28.235.254:34501)
0926 07:19:47.653174 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:47.653189 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:47.653224 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:19:47.653905 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:47.653923 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:47.654242 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:47.654523 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:47.655375 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:47.656517 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:47.657748 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:47.658329 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:47.658474 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:47.661723 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:47.661729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:47.661742 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:47.661770 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:47.663905 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:47.664451 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:47.664457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:47.664463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:47.664618 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:47.664967 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:47.664970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:47.664972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:47.665563 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:47.666463 (+ 900us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:47.666666 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 07:19:47.666998 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":157,"threads_started":1}
I20260926 07:19:47.669706 3293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:47.675763 3293 catalog_manager.cc:1347] Generated new cluster ID: 2d2fff5f73d349f7a0f0a40ca3672ad4
I20260926 07:19:47.675835 3293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:47.689005 3293 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:47.690326 3293 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:47.699028 3293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496: Generated new TSK 0
I20260926 07:19:47.699870 3293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab.
Entry for principal oryx/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab.
I20260926 07:19:47.741513 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34501
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab
--principal=oryx/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:47.854913 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:47.855175 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:47.855217 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:47.859930 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:47.859993 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:47.860085 3305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:47.865643 3305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab
--principal=oryx/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:47.866771 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:47.867866 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:19:47.878149 3305 server_base.cc:1034] running on GCE node
W20260926 07:19:47.878084 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:47.878084 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:47.878299 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:47.878957 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:47.879719 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:47.880903 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790407187880872 us; error 42 us; skew 500 ppm
Sep 26 07:19:47 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407187, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:47.884019 3305 init.cc:377] Logged in from keytab as oryx/127.28.235.193@KRBTEST.COM (short username oryx)
I20260926 07:19:47.884891 3305 webserver.cc:460] Webserver started at http://127.28.235.193:35725/ using document root <none> and password file <none>
I20260926 07:19:47.885528 3305 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:47.885598 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:47.885774 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:47.887964 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "76925f7ba8f842d1a98627db55b76657"
format_stamp: "Formatted at 2026-09-26 07:19:47 on dist-test-slave-7zlw"
server_key: "2c25a7f9bab9f15cf5c570c931f71efa"
server_key_iv: "5d8d23f914ebaf543e17eee06c678b70"
server_key_version: "encryptionkey@0"
I20260926 07:19:47.888559 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "76925f7ba8f842d1a98627db55b76657"
format_stamp: "Formatted at 2026-09-26 07:19:47 on dist-test-slave-7zlw"
server_key: "2c25a7f9bab9f15cf5c570c931f71efa"
server_key_iv: "5d8d23f914ebaf543e17eee06c678b70"
server_key_version: "encryptionkey@0"
I20260926 07:19:47.892838 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:19:47.895617 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:47.897229 3305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:47.897375 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "76925f7ba8f842d1a98627db55b76657"
format_stamp: "Formatted at 2026-09-26 07:19:47 on dist-test-slave-7zlw"
server_key: "2c25a7f9bab9f15cf5c570c931f71efa"
server_key_iv: "5d8d23f914ebaf543e17eee06c678b70"
server_key_version: "encryptionkey@0"
I20260926 07:19:47.897497 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:47.910926 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:47.914363 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:47.914587 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:47.915256 3305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:47.916198 3305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:47.916266 3305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:47.916337 3305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:47.916388 3305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:47.925602 3305 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:35801
I20260926 07:19:47.925630 3433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:35801 every 8 connection(s)
I20260926 07:19:47.926493 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:47.928175 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 3305
I20260926 07:19:47.928331 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:47.928686 29615 external_mini_cluster.cc:1428] Setting key 060f8dd39093db76dfef5ae31bdd34d0
WARNING: no policy specified for oryx/127.28.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.194@KRBTEST.COM" created.
Sep 26 07:19:47 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407187, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:47.947333 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:47.928392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:45021 (local address 127.28.235.254:34501)
0926 07:19:47.928560 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:47.928565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:47.929683 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 07:19:47.931681 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:47.931683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:47.931728 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:47.931793 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:47.933159 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:47.933811 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:47.935027 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:47.935286 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:47.935406 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:47.943525 (+ 8119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:47.943535 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:47.943537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:47.943566 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:47.945138 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:47.945703 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:47.945706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:47.945708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:47.945788 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:47.946158 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:47.946161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:47.946163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:47.946360 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:47.946485 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:47.947038 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 07:19:47.947168 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:19:47.948217 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:47.928858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.193:45021)
0926 07:19:47.929499 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:47.929556 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:47.931409 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:47.931997 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:47.932009 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:47.932471 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:47.932974 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:47.932988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:47.933950 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:47.933953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:47.934893 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:47.934902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:47.935446 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:47.935450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:47.935661 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:47.940768 (+ 5107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:47.940788 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:47.943275 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:47.945274 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:47.945281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:47.945300 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:47.945585 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:47.945883 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:47.945886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:47.945888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:47.946049 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:47.946485 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:47.946494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:47.946769 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 07:19:47.946984 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 07:19:47.947339 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":225,"threads_started":1}
I20260926 07:19:47.950196 3434 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34501
I20260926 07:19:47.950475 3434 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:47.951066 3434 heartbeater.cc:511] Master 127.28.235.254:34501 requested a full tablet report, sending...
I20260926 07:19:47.953182 3236 ts_manager.cc:194] Registered new tserver with Master: 76925f7ba8f842d1a98627db55b76657 (127.28.235.193:35801)
I20260926 07:19:47.954737 3236 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:45021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab.
Entry for principal oryx/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab.
I20260926 07:19:47.961164 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34501
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab
--principal=oryx/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:48.076351 3439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:48.076658 3439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:48.076704 3439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:48.080857 3439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:48.080932 3439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:48.080996 3439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:48.086493 3439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab
--principal=oryx/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:48.087586 3439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:48.088573 3439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:48.098512 3447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:48.098500 3444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:48.098769 3439 server_base.cc:1034] running on GCE node
W20260926 07:19:48.098500 3445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:48.099328 3439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:48.099994 3439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:48.101106 3439 hybrid_clock.cc:648] HybridClock initialized: now 1790407188101085 us; error 30 us; skew 500 ppm
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:48.103952 3439 init.cc:377] Logged in from keytab as oryx/127.28.235.194@KRBTEST.COM (short username oryx)
I20260926 07:19:48.104812 3439 webserver.cc:460] Webserver started at http://127.28.235.194:36839/ using document root <none> and password file <none>
I20260926 07:19:48.105419 3439 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:48.105481 3439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:48.105720 3439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:48.107759 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "89e80c04d6c24d0983e9a0d4f71ae59f"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "f92ad4c0811eb0ffa65f1a7f05200b84"
server_key_iv: "d78291bb3cccf89da5acebef4bca8a93"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.108286 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "89e80c04d6c24d0983e9a0d4f71ae59f"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "f92ad4c0811eb0ffa65f1a7f05200b84"
server_key_iv: "d78291bb3cccf89da5acebef4bca8a93"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.112342 3439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:19:48.115058 3454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:48.116732 3439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:19:48.116856 3439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "89e80c04d6c24d0983e9a0d4f71ae59f"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "f92ad4c0811eb0ffa65f1a7f05200b84"
server_key_iv: "d78291bb3cccf89da5acebef4bca8a93"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.117072 3439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:48.148391 3439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:48.152261 3439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:48.152485 3439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:48.153169 3439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:48.154047 3439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:48.154120 3439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:48.154193 3439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:48.154244 3439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:48.163472 3439 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:41571
I20260926 07:19:48.163496 3567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:41571 every 8 connection(s)
I20260926 07:19:48.164403 3439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:48.168220 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 3439
I20260926 07:19:48.168351 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:48.168645 29615 external_mini_cluster.cc:1428] Setting key d300feeaab349ad58c7530552f0a21ae
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:19:48.181159 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.166591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:38425 (local address 127.28.235.254:34501)
0926 07:19:48.166722 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.166727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.167809 (+ 1082us) server_negotiation.cc:407] Connection header received
0926 07:19:48.169569 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.169572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.169614 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.169687 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.171180 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.172020 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.173528 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.173759 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.173861 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.177000 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.177004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.177006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.177078 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.178814 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.179416 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.179420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.179422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.179520 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.179911 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.179915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.179917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.180123 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.180241 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.180791 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.180937 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.195@KRBTEST.COM" created.
I20260926 07:19:48.182308 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.167130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.194:38425)
0926 07:19:48.167644 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.167691 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.169315 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.169898 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.169909 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.170371 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.170992 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.171015 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.172159 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.172167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.173391 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.173406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.173938 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.173945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.174191 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.174883 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.174917 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.176796 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:48.178970 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.178977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.178996 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.179297 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.179630 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.179635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.179637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.179794 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.180240 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:48.180250 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:48.180530 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 07:19:48.180746 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 07:19:48.181155 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":107,"threads_started":1}
I20260926 07:19:48.183821 3568 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34501
I20260926 07:19:48.184129 3568 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:48.184697 3568 heartbeater.cc:511] Master 127.28.235.254:34501 requested a full tablet report, sending...
I20260926 07:19:48.185976 3236 ts_manager.cc:194] Registered new tserver with Master: 89e80c04d6c24d0983e9a0d4f71ae59f (127.28.235.194:41571)
I20260926 07:19:48.186625 3236 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:38425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab.
Entry for principal oryx/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab.
I20260926 07:19:48.200027 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34501
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab
--principal=oryx/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:48.318845 3573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:48.319183 3573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:48.319233 3573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:48.323585 3573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:48.323665 3573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:48.323732 3573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:48.329288 3573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab
--principal=oryx/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:48.330435 3573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:48.331439 3573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:48.341602 3581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:48.341595 3578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:48.341928 3579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:48.341701 3573 server_base.cc:1034] running on GCE node
I20260926 07:19:48.342366 3573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:48.343078 3573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:48.344314 3573 hybrid_clock.cc:648] HybridClock initialized: now 1790407188344293 us; error 30 us; skew 500 ppm
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:48.347383 3573 init.cc:377] Logged in from keytab as oryx/127.28.235.195@KRBTEST.COM (short username oryx)
I20260926 07:19:48.348210 3573 webserver.cc:460] Webserver started at http://127.28.235.195:36481/ using document root <none> and password file <none>
I20260926 07:19:48.348768 3573 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:48.348829 3573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:48.349002 3573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:48.351298 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "ab96d3b6385040429868d9fc330d6b92"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "33055e72b90e27d1a6c6334a117962ac"
server_key_iv: "e42b929f528a1becd1d176cc8a4db163"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.351836 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "ab96d3b6385040429868d9fc330d6b92"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "33055e72b90e27d1a6c6334a117962ac"
server_key_iv: "e42b929f528a1becd1d176cc8a4db163"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.355839 3573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 07:19:48.358553 3588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:48.359910 3573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:19:48.360076 3573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "ab96d3b6385040429868d9fc330d6b92"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "33055e72b90e27d1a6c6334a117962ac"
server_key_iv: "e42b929f528a1becd1d176cc8a4db163"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.360198 3573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:48.371685 3573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:48.375101 3573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:48.375308 3573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:48.375909 3573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:48.376775 3573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:48.376821 3573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:48.376863 3573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:48.376916 3573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:48.386389 3573 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:45051
I20260926 07:19:48.386420 3701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:45051 every 8 connection(s)
I20260926 07:19:48.387437 3573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:48.397214 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 3573
I20260926 07:19:48.397359 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:48.397652 29615 external_mini_cluster.cc:1428] Setting key 192f745893240dfb8cec19603b534886
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:48.403132 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.389673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:33459 (local address 127.28.235.254:34501)
0926 07:19:48.389873 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.389879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.390726 (+ 847us) server_negotiation.cc:407] Connection header received
0926 07:19:48.392320 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.392324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.392399 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.392518 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.393861 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.394459 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.395706 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.395981 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.396109 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.399072 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.399083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.399087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.399114 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.400702 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.401459 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.401465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.401467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.401561 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.401942 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.401946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.401948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.402145 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.402258 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.402809 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.402939 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 07:19:48.403946 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.390080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.195:33459)
0926 07:19:48.390561 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.390607 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.392083 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.392699 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.392707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.393176 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.393697 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.393712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.394597 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.394602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.395554 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.395564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.396149 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.396154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.396340 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.397094 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.397125 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.398893 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:48.400864 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.400871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.400895 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.401349 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.401663 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.401668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.401670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.401825 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.402267 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:48.402276 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:48.402554 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 07:19:48.402750 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 07:19:48.403094 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":103,"threads_started":1}
I20260926 07:19:48.405148 3702 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34501
I20260926 07:19:48.405436 3702 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:48.405973 3702 heartbeater.cc:511] Master 127.28.235.254:34501 requested a full tablet report, sending...
I20260926 07:19:48.407104 3236 ts_manager.cc:194] Registered new tserver with Master: ab96d3b6385040429868d9fc330d6b92 (127.28.235.195:45051)
I20260926 07:19:48.407647 3236 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:33459
I20260926 07:19:48.412056 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.235.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.253.keytab.
Entry for principal oryx/127.28.235.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.253.keytab.
I20260926 07:19:48.444006 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.253:36465
--webserver_interface=127.28.235.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.253.keytab
--principal=oryx/127.28.235.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:34501,127.28.235.253:36465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:48.556368 3707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:48.556694 3707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:48.556777 3707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:48.560863 3707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:48.560978 3707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:48.561014 3707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:48.561065 3707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:48.561124 3707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:48.567090 3707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:34501,127.28.235.253:36465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.253.keytab
--principal=oryx/127.28.235.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.253:36465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.235.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
W20260926 07:19:48.567378 3707 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.235.254:34501,127.28.235.253:36465'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 07:19:48.568334 3707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:48.569475 3707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:19:48.578797 3707 server_base.cc:1034] running on GCE node
W20260926 07:19:48.578797 3712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:48.578761 3715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:48.578814 3713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:48.579881 3707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:48.581203 3707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:48.582396 3707 hybrid_clock.cc:648] HybridClock initialized: now 1790407188582378 us; error 37 us; skew 500 ppm
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:48.585518 3707 init.cc:377] Logged in from keytab as oryx/127.28.235.253@KRBTEST.COM (short username oryx)
I20260926 07:19:48.586503 3707 webserver.cc:460] Webserver started at http://127.28.235.253:35501/ using document root <none> and password file <none>
I20260926 07:19:48.587132 3707 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:48.587237 3707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:48.587494 3707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:48.589784 3707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data/instance:
uuid: "ec96a06c50974233bbfdd1b858a3d28c"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "4494fdce27d0575dd4c5a19c6c1d510f"
server_key_iv: "debee22f767ab0d81627a135d6ded912"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.590399 3707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal/instance:
uuid: "ec96a06c50974233bbfdd1b858a3d28c"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "4494fdce27d0575dd4c5a19c6c1d510f"
server_key_iv: "debee22f767ab0d81627a135d6ded912"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.594574 3707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:48.597308 3722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:48.598842 3707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:48.599025 3707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal
uuid: "ec96a06c50974233bbfdd1b858a3d28c"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "4494fdce27d0575dd4c5a19c6c1d510f"
server_key_iv: "debee22f767ab0d81627a135d6ded912"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.599159 3707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:48.615794 3707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:48.619238 3707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:48.619462 3707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:48.626739 3707 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.253:36465
I20260926 07:19:48.627194 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.253:36465 every 8 connection(s)
I20260926 07:19:48.627769 3707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
I20260926 07:19:48.630739 3775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:48.630949 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 3707
I20260926 07:19:48.635773 3775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.235.253@KRBTEST.COM
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.253@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:48.646185 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.632460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.28.235.253:36465)
0926 07:19:48.632916 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.632931 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.632965 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:19:48.633717 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.633736 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.634068 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.634400 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.635188 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.636216 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.637127 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.637742 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.637897 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.640934 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.640940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.640952 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.640982 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.642997 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.643512 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.643519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.643525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.643681 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.643982 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.643985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.643987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.644475 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.644661 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.644953 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.645238 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":150,"threads_started":1}
I20260926 07:19:48.647337 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.637491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.28.235.254:34501)
0926 07:19:48.637644 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.637648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.637962 (+ 314us) server_negotiation.cc:407] Connection header received
0926 07:19:48.638167 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.638170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.638222 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.638306 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.638995 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.639831 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.640843 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.641142 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.641557 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.643653 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.643656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.643659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.643683 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.645606 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.646071 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.646075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.646077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.646156 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.646450 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.646453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.646455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.646652 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.646764 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.647082 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.647197 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:19:48.648358 3779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.637439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42076)
0926 07:19:48.637852 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.637900 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.638045 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.638350 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.638356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.638712 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.638829 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.638835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.639983 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.639991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.640724 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.640733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.641266 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.641271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.641426 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.641985 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.642005 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.643460 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:48.645723 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.645731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.645737 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.645967 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.646250 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.646253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.646255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.646340 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.646760 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:48.646766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:48.646879 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 07:19:48.647026 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:19:48.647197 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":149,"threads_started":1}
I20260926 07:19:48.649529 3775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.253@KRBTEST.COM for oryx/127.28.235.253@KRBTEST.COM
I20260926 07:19:48.658386 3779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.650006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36465 (local address 127.0.0.1:57172)
0926 07:19:48.650150 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.650162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.650223 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.650494 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.650497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.650620 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.650709 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.650714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.652145 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.652148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.652652 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.652660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.653278 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.653282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.653380 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.653913 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.653932 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.655176 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:48.656907 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.656910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.656912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.657198 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.657490 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.657493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.657494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.657572 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.657899 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:48.657901 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:48.658043 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 07:19:48.658148 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:19:48.658249 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 07:19:48.658419 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.650079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.28.235.253:36465)
0926 07:19:48.650327 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.650330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.650339 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:19:48.650362 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.650365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.650394 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.650453 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.650837 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.652020 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.652756 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.653139 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.653235 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.655323 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.655326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.655328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.655355 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.656795 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.657301 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.657304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.657305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.657381 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.657642 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.657643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.657645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.657799 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.657869 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.658204 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.658293 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260926 07:19:48.661336 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: Bootstrap starting.
I20260926 07:19:48.663726 3775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:48.664559 3775 log.cc:824] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:48.666813 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: No bootstrap required, opened a new log
I20260926 07:19:48.670056 3775 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } }
I20260926 07:19:48.670277 3775 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:48.670312 3775 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec96a06c50974233bbfdd1b858a3d28c, State: Initialized, Role: FOLLOWER
I20260926 07:19:48.670843 3775 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } }
I20260926 07:19:48.671366 3781 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } } }
I20260926 07:19:48.671486 3781 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:19:48.672202 3775 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:19:48.677255 3792 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:19:48.677347 3792 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:19:48.679736 3707 master_runner.cc:428] Detected that this master ec96a06c50974233bbfdd1b858a3d28c is joining an existing cluster
I20260926 07:19:48.679809 3707 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.235.253:36465
I20260926 07:19:48.690212 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.682044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.28.235.253:36465)
0926 07:19:48.682161 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.682166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.682520 (+ 354us) server_negotiation.cc:407] Connection header received
0926 07:19:48.682657 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.682660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.682705 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.682786 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.683256 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.683859 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.685091 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.685341 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.685463 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.686799 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.686802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.686805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.686827 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.688503 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.688950 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.688953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.688955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.689079 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.689368 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.689373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.689377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.689553 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.689674 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.689906 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.690059 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":39}
I20260926 07:19:48.690225 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.682031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36465 (local address 127.0.0.1:57176)
0926 07:19:48.682441 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.682458 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.682556 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.682815 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.682818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.683028 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.683141 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.683147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.683989 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.683992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.684948 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.684958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.685468 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.685471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.685600 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.686195 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.686215 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.686670 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:48.688627 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.688631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.688633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.688819 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.689168 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.689174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.689179 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.689267 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.689659 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:48.689662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:48.689741 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 07:19:48.689857 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 07:19:48.690008 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":44,"thread_start_us":89,"threads_started":1}
I20260926 07:19:48.690526 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.682412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.28.235.254:34501)
0926 07:19:48.682542 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.682549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.682564 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:19:48.682907 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.682921 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.682971 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.683047 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.683642 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.684269 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.685201 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.685548 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.685652 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.687110 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.687115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.687118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.687151 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.688775 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.689234 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.689237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.689239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.689318 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.689609 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.689612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.689614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.689838 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.689908 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.690246 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.690363 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:19:48.691061 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.681970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42082)
0926 07:19:48.682442 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.682458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.682556 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.683160 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.683164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.683383 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.683514 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.683521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.684379 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.684385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.684995 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.685003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.685775 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.685779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.685887 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.686355 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.686373 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.686936 (+ 563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:48.688880 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.688886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.688888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.689152 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.689422 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.689425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.689427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.689510 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.689989 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:48.689992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:48.690076 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 07:19:48.690739 (+ 663us) client_negotiation.cc:240] Negotiation successful
0926 07:19:48.690877 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":23,"spinlock_wait_cycles":384,"thread_start_us":106,"threads_started":1}
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407187, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.254@KRBTEST.COM for oryx/127.28.235.253@KRBTEST.COM
I20260926 07:19:48.705775 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.695610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36465 (local address 127.0.0.1:57186)
0926 07:19:48.696008 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.696058 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.696177 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.696571 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.696577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.696951 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.697121 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.697128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.697838 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.697842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.698503 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.698512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.699123 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.699126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.699252 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.699963 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.699975 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.701671 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:48.703681 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.703691 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.703697 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.704200 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.704491 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.704494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.704496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.704579 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.705101 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:48.705111 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:48.705240 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 07:19:48.705386 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 07:19:48.705588 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20260926 07:19:48.706290 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.695636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.28.235.253:36465)
0926 07:19:48.696298 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.696303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.696315 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:19:48.696375 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.696377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.696439 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.696532 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.697233 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.697728 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.698653 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.699002 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.699117 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.701826 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.701831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.701835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.701861 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.703556 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.704314 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.704318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.704319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.704396 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.704668 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.704671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.704673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.704924 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.705045 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.706033 (+ 988us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.706146 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581}
I20260926 07:19:48.710639 3236 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.235.253:36465: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c"
member_type: NON_VOTER
last_known_addr {
host: "127.28.235.253"
port: 36465
}
attrs {
promote: true
}
}
dest_uuid: "3335d929499c4a7dae82c7ae2aa00496"
cas_config_opid_index: -1
I20260926 07:19:48.711304 3236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.28.235.252@KRBTEST.COM; defaulting to no policy
I20260926 07:19:48.713311 3803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ec96a06c50974233bbfdd1b858a3d28c (127.28.235.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } } }
I20260926 07:19:48.714016 3747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:48.716046 3803 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.235.253:36465
I20260926 07:19:48.716158 3747 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Refusing update from remote peer 3335d929499c4a7dae82c7ae2aa00496: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.252@KRBTEST.COM" created.
I20260926 07:19:48.716527 3707 master.cc:537] Master@127.28.235.253:36465 shutting down...
I20260926 07:19:48.719022 3803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:19:48.721724 3230 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 07:19:48.719348 3804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } } }
I20260926 07:19:48.722316 3804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:48.722556 3707 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 07:19:48.722718 3230 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 -> Peer ec96a06c50974233bbfdd1b858a3d28c (127.28.235.253:36465): Couldn't send request to peer ec96a06c50974233bbfdd1b858a3d28c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:19:48.722900 3707 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:48.723033 3707 tablet_replica.cc:330] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: stopping tablet replica
I20260926 07:19:48.728894 3707 master.cc:552] Master@127.28.235.253:36465 shutdown complete.
I20260926 07:19:48.728961 3707 master_runner.cc:305] Clearing existing system tablet
I20260926 07:19:48.729275 3707 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:48.729333 3707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:48.730711 3707 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:48.731930 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:48.732162 3707 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:48.732311 3707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:48.732381 3707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal
uuid: "ec96a06c50974233bbfdd1b858a3d28c"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "4494fdce27d0575dd4c5a19c6c1d510f"
server_key_iv: "debee22f767ab0d81627a135d6ded912"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.734465 3707 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.252.keytab.
Entry for principal oryx/127.28.235.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.252.keytab.
I20260926 07:19:48.737007 3707 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:19:48.737054 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.252:39391
--webserver_interface=127.28.235.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.252.keytab
--principal=oryx/127.28.235.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:34501,127.28.235.253:36465,127.28.235.252:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:19:48.737186 3707 log.cc:1196] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 07:19:48.737459 3707 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: Deleting consensus metadata
I20260926 07:19:48.737651 3707 master_runner.cc:315] Copying system tablet from 127.28.235.254:34501
I20260926 07:19:48.738935 3707 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.235.254:34501
I20260926 07:19:48.747530 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.739614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.28.235.254:34501)
0926 07:19:48.739759 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.739763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.739995 (+ 232us) server_negotiation.cc:407] Connection header received
0926 07:19:48.740148 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.740157 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.740205 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.740296 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.740754 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.741421 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.742247 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.742483 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.742597 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.744230 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.744233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.744236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.744269 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.745797 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.746255 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.746259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.746262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.746345 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.746645 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.746650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.746652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.746845 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.746963 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.747235 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.747354 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:19:48.747545 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.739533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42098)
0926 07:19:48.739889 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.739909 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.740043 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.740314 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.740317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.740529 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.740628 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.740634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.741538 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.741543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.742124 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.742133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.742603 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.742606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.742709 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.743670 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.743686 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.744079 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:48.745929 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.745933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.745935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.746150 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.746450 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.746454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.746456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.746551 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.746950 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:48.746953 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:48.747062 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 07:19:48.747167 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:19:48.747322 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":85,"threads_started":1}
I20260926 07:19:48.748492 3263 tablet_copy_service.cc:140] P 3335d929499c4a7dae82c7ae2aa00496: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ec96a06c50974233bbfdd1b858a3d28c ({username='oryx', principal='oryx/127.28.235.253@KRBTEST.COM'} at 127.0.0.1:42098)
I20260926 07:19:48.748687 3263 tablet_copy_service.cc:161] P 3335d929499c4a7dae82c7ae2aa00496: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ec96a06c50974233bbfdd1b858a3d28c at {username='oryx', principal='oryx/127.28.235.253@KRBTEST.COM'} at 127.0.0.1:42098: session id = ec96a06c50974233bbfdd1b858a3d28c-00000000000000000000000000000000
I20260926 07:19:48.750236 3263 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:19:48.751824 3707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:19:48.753921 3707 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: tablet copy: Starting download of 0 data blocks...
I20260926 07:19:48.754136 3707 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: tablet copy: Starting download of 1 WAL segments...
I20260926 07:19:48.755939 3707 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:19:48.757699 3263 tablet_copy_service.cc:342] P 3335d929499c4a7dae82c7ae2aa00496: Request end of tablet copy session ec96a06c50974233bbfdd1b858a3d28c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.235.253@KRBTEST.COM'} at 127.0.0.1:42098
I20260926 07:19:48.757836 3263 tablet_copy_service.cc:433] P 3335d929499c4a7dae82c7ae2aa00496: ending tablet copy session ec96a06c50974233bbfdd1b858a3d28c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ec96a06c50974233bbfdd1b858a3d28c
I20260926 07:19:48.765121 3707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:48.767174 3818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:48.767225 3821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:48.769310 3707 server_base.cc:1034] running on GCE node
W20260926 07:19:48.769474 3819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:48.769701 3707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:48.770140 3707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:48.771301 3707 hybrid_clock.cc:648] HybridClock initialized: now 1790407188771287 us; error 36 us; skew 500 ppm
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:48.772879 3707 init.cc:377] Logged in from keytab as oryx/127.28.235.253@KRBTEST.COM (short username oryx)
I20260926 07:19:48.773449 3707 webserver.cc:460] Webserver started at http://127.28.235.253:34595/ using document root <none> and password file <none>
I20260926 07:19:48.773659 3707 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:48.773713 3707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:48.775040 3707 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:48.776006 3828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:48.776229 3707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:48.776321 3707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal
uuid: "ec96a06c50974233bbfdd1b858a3d28c"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "4494fdce27d0575dd4c5a19c6c1d510f"
server_key_iv: "debee22f767ab0d81627a135d6ded912"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.776430 3707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:48.782568 3707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:48.784608 3707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:48.789760 3707 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.253:36465
I20260926 07:19:48.789779 3880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.253:36465 every 8 connection(s)
I20260926 07:19:48.790416 3707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
I20260926 07:19:48.794314 3881 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:19:48.795181 3881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: Bootstrap starting.
I20260926 07:19:48.804220 3881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:19:48.804574 3881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: Bootstrap complete.
I20260926 07:19:48.805161 3881 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } }
I20260926 07:19:48.805299 3881 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec96a06c50974233bbfdd1b858a3d28c, State: Initialized, Role: LEARNER
I20260926 07:19:48.805447 3881 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } }
I20260926 07:19:48.805811 3884 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } } }
I20260926 07:19:48.805977 3884 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: This master's current role is: LEARNER
I20260926 07:19:48.805984 3881 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:48.809365 3895 catalog_manager.cc:1259] Loaded cluster ID: 2d2fff5f73d349f7a0f0a40ca3672ad4
I20260926 07:19:48.809437 3895 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: loading cluster ID for follower catalog manager: success
I20260926 07:19:48.810968 3895 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: acquiring CA information for follower catalog manager: success
I20260926 07:19:48.812100 3895 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:19:48.891042 3814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:48.891316 3814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:48.891356 3814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:48.895666 3814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:48.895741 3814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:48.895771 3814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:48.895792 3814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:48.895818 3814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:48.901490 3814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:34501,127.28.235.253:36465,127.28.235.252:39391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.252.keytab
--principal=oryx/127.28.235.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.252:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.235.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:48.902678 3814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:48.903746 3814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:48.913125 3901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:48.913174 3903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:48.913206 3900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:48.913342 3814 server_base.cc:1034] running on GCE node
I20260926 07:19:48.914031 3814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:48.915307 3814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:48.916518 3814 hybrid_clock.cc:648] HybridClock initialized: now 1790407188916499 us; error 49 us; skew 500 ppm
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:48.919611 3814 init.cc:377] Logged in from keytab as oryx/127.28.235.252@KRBTEST.COM (short username oryx)
I20260926 07:19:48.920653 3814 webserver.cc:460] Webserver started at http://127.28.235.252:44369/ using document root <none> and password file <none>
I20260926 07:19:48.921463 3814 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:48.921576 3814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:48.921792 3814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:48.923902 3814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data/instance:
uuid: "0259221937c64b3188c7fdcf29a0efc7"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "281d1314bfd7cd3bde0009610ea27492"
server_key_iv: "c06a201ecefb52edfbac0443b7ddf613"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.924499 3814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal/instance:
uuid: "0259221937c64b3188c7fdcf29a0efc7"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "281d1314bfd7cd3bde0009610ea27492"
server_key_iv: "c06a201ecefb52edfbac0443b7ddf613"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.928334 3814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:19:48.930971 3910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:48.932291 3814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:19:48.932446 3814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal
uuid: "0259221937c64b3188c7fdcf29a0efc7"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "281d1314bfd7cd3bde0009610ea27492"
server_key_iv: "c06a201ecefb52edfbac0443b7ddf613"
server_key_version: "encryptionkey@0"
I20260926 07:19:48.932596 3814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:48.948992 3814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:48.952500 3814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:48.952786 3814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:48.957531 3434 heartbeater.cc:503] Master 127.28.235.254:34501 was elected leader, sending a full tablet report...
I20260926 07:19:48.960311 3814 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.252:39391
I20260926 07:19:48.960601 3962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.252:39391 every 8 connection(s)
I20260926 07:19:48.961382 3814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
I20260926 07:19:48.964268 3963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:48.965411 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 3814
I20260926 07:19:48.968735 3963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.235.252@KRBTEST.COM
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.252@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:48.981171 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.970695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.28.235.254:34501)
0926 07:19:48.970828 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.970833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.971069 (+ 236us) server_negotiation.cc:407] Connection header received
0926 07:19:48.971258 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.971261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.971303 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.971389 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.972050 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.972581 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.973558 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.973784 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.973895 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.977139 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.977142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.977144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.977170 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.978952 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.979543 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.979546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.979547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.979644 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.979979 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.979982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.979983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.980282 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.980397 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.980880 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.980979 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:19:48.982198 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.970461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42104)
0926 07:19:48.970890 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.970930 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.971140 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.971437 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.971443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.971802 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.971917 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.971923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.972724 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.972727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.973440 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.973449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.973933 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.973936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.974050 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.974697 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.974716 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.976898 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:48.979066 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.979073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.979084 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.979390 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.979752 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.979755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.979757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.979845 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.980482 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:48.980489 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:48.980615 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 07:19:48.980806 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 07:19:48.981149 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":143,"thread_start_us":132,"threads_started":1}
I20260926 07:19:48.982198 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.967007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.28.235.252:39391)
0926 07:19:48.967527 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.967543 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.967582 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:19:48.968286 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.968305 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.968666 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.969073 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.969890 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.971562 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.972514 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.973117 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.973303 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.976483 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.976491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.976506 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.976534 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.978365 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.978903 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.978909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.978915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.979088 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.979370 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.979375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.979377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.980013 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.980207 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.980613 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.981082 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":160,"threads_started":1}
I20260926 07:19:48.983559 3963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.252@KRBTEST.COM for oryx/127.28.235.253@KRBTEST.COM
I20260926 07:19:48.992002 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.984080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36465 (local address 127.0.0.1:57200)
0926 07:19:48.984318 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.984337 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.984428 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.984824 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.984826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.984958 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.985111 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.985117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.986254 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.986256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.986777 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.986784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.987246 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.987248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.987327 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.987785 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.987796 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.989097 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:48.990697 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.990701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.990704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.990935 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.991196 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:48.991199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:48.991200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:48.991283 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:48.991631 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:48.991633 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:48.991705 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 07:19:48.991782 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:19:48.991875 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260926 07:19:48.992053 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.984207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.28.235.253:36465)
0926 07:19:48.984619 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.984623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.984635 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:19:48.984673 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.984675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.984717 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.984792 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.985229 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.986139 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.986914 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.987147 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.987236 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.989246 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.989249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.989251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.989274 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:48.990580 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.991029 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.991033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.991034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.991117 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:48.991359 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:48.991361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:48.991363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:48.991540 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:48.991615 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:48.991836 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 07:19:48.991932 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":98,"threads_started":1}
I20260926 07:19:48.992900 3963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:19:48 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407188, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.252@KRBTEST.COM for oryx/127.28.235.252@KRBTEST.COM
I20260926 07:19:49.002202 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.993430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:39391 (local address 127.0.0.1:47810)
0926 07:19:48.993568 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:48.993582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:48.993664 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:48.993900 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:48.993902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:48.994070 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:48.994131 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.994136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.995236 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.995239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.995738 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:48.995745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.996185 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.996188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:48.996260 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.996676 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:48.996688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:48.998438 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.000305 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.000309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.000311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.000580 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.000937 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.000942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.000945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.001067 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.001608 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.001613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.001721 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.001858 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.001996 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 07:19:49.002202 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:48.993522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.28.235.252:39391)
0926 07:19:48.993605 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:48.993611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:48.993626 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:19:48.993765 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:48.993770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:48.993810 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:48.993892 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:48.994235 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.995109 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.995846 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:48.996083 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:48.996189 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:48.998596 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:48.998599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:48.998602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:48.998650 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.000194 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.000699 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.000702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.000704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.000831 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.001171 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.001174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.001176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.001453 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.001548 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.001914 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.002048 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":25}
I20260926 07:19:49.005834 3963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Bootstrap starting.
I20260926 07:19:49.008658 3963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:49.009467 3963 log.cc:824] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:49.011399 3963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: No bootstrap required, opened a new log
I20260926 07:19:49.014468 3963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } }
I20260926 07:19:49.014715 3963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:49.014803 3963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0259221937c64b3188c7fdcf29a0efc7, State: Initialized, Role: FOLLOWER
I20260926 07:19:49.015328 3963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } }
I20260926 07:19:49.015894 3969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } } }
I20260926 07:19:49.016053 3969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:19:49.016609 3963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:19:49.021445 3980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:19:49.021550 3980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:19:49.025507 3814 master_runner.cc:428] Detected that this master 0259221937c64b3188c7fdcf29a0efc7 is joining an existing cluster
I20260926 07:19:49.025581 3814 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.235.252:39391
I20260926 07:19:49.036154 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 3305
I20260926 07:19:49.038697 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.028177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36465 (local address 127.0.0.1:57212)
0926 07:19:49.028795 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.028811 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.028950 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.029373 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.029376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.029622 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.029780 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.029789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.030885 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.030888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.031894 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.031909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.032653 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.032658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.033205 (+ 547us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.033772 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.033789 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.034266 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.036872 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.036875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.036877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.037147 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.037552 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.037555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.037558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.037658 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.038103 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.038105 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.038215 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.038309 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.038478 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":102,"spinlock_wait_cycles":1792,"thread_start_us":423,"threads_started":1}
I20260926 07:19:49.038733 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.028239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.28.235.253:36465)
0926 07:19:49.028357 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.028362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.028879 (+ 517us) server_negotiation.cc:407] Connection header received
0926 07:19:49.029086 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.029089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.029156 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.029217 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.029918 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.030745 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.032036 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.032393 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.032471 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.034437 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.034440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.034442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.034465 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.036758 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.037301 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.037305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.037307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.037420 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.037755 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.037758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.037760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.037966 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.038046 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.038467 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.038610 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:19:49.042676 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:35801
--local_ip_for_outbound_sockets=127.28.235.193
--tserver_master_addrs=127.28.235.254:34501
--webserver_port=35725
--webserver_interface=127.28.235.193
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab
--principal=oryx/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.235.254:34501,127.28.235.253:36465
--tserver_master_addrs=127.28.235.254:34501,127.28.235.253:36465,127.28.235.252:39391 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:19:49.066145 3988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.029153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:39391 (local address 127.0.0.1:47824)
0926 07:19:49.029517 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.029534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.029678 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.030003 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.030006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.030218 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.030327 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.030334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.031344 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.031347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.032389 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.032399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.033036 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.033044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.033255 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.061737 (+ 28482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.061772 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.062385 (+ 613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.064457 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.064460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.064462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.064711 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.065060 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.065062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.065064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.065139 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.065597 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.065600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.065689 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.065773 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.065939 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":29,"spinlock_wait_cycles":8320,"thread_start_us":80,"threads_started":1}
I20260926 07:19:49.066489 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.028323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.28.235.254:34501)
0926 07:19:49.028472 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.028479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.029076 (+ 597us) server_negotiation.cc:407] Connection header received
0926 07:19:49.029286 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.029290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.029355 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.029416 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.030166 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.031081 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.032678 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.033012 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.033127 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.062691 (+ 29564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.062696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.062699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.062733 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.064563 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.065095 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.065099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.065101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.065193 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.065507 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.065510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.065512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.065760 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.065844 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.066194 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.066319 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:19:49.066753 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.029036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.28.235.252:39391)
0926 07:19:49.029146 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.029150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.029594 (+ 444us) server_negotiation.cc:407] Connection header received
0926 07:19:49.029779 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.029784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.029828 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.029879 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.030454 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.031228 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.032528 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.032893 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.032992 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.062603 (+ 29611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.062607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.062610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.062631 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.064319 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.064867 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.064870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.064872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.064944 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.065225 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.065227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.065229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.065440 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.065540 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.066507 (+ 967us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.066625 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":30}
I20260926 07:19:49.068118 3987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.028313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42114)
0926 07:19:49.028951 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.028968 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.029150 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.029591 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.029593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.029872 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.030023 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.030031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.031218 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.031221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.032559 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.032567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.033299 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.033303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.033435 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.061780 (+ 28345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.061799 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.062471 (+ 672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.064704 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.064707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.064709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.064940 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.065306 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.065309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.065311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.065392 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.065902 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.065905 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.066014 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.067761 (+ 1747us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.067906 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"spinlock_wait_cycles":438528,"thread_start_us":289,"threads_started":1}
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407187, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.254@KRBTEST.COM for oryx/127.28.235.252@KRBTEST.COM
I20260926 07:19:49.081909 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.072444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.28.235.252:39391)
0926 07:19:49.072576 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.072582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.072607 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:19:49.072674 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.072677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.072727 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.072812 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.073439 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.074207 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.075352 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.075691 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.075800 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.078285 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.078289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.078292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.078320 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.080031 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.080549 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.080553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.080554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.080642 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.080961 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.080965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.080966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.081211 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.081312 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.081625 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.081747 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:19:49.081909 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.072191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:39391 (local address 127.0.0.1:47836)
0926 07:19:49.072433 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.072459 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.072539 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.072848 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.072852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.073203 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.073318 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.073326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.074335 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.074338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.075221 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.075232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.075820 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.075824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.075927 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.076476 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.076492 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.078128 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.080166 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.080169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.080172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.080438 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.080744 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.080746 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.080748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.080852 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.081318 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.081321 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.081444 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.081558 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.081724 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260926 07:19:49.083367 3236 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.235.252:39391: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7"
member_type: NON_VOTER
last_known_addr {
host: "127.28.235.252"
port: 39391
}
attrs {
promote: true
}
}
dest_uuid: "3335d929499c4a7dae82c7ae2aa00496"
cas_config_opid_index: 5
I20260926 07:19:49.083983 3236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } }
I20260926 07:19:49.085247 3803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0259221937c64b3188c7fdcf29a0efc7 (127.28.235.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.086136 3937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:49.086794 3803 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.235.252:39391
I20260926 07:19:49.086974 3803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.087057 3803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:49.087733 3937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 FOLLOWER]: Refusing update from remote peer 3335d929499c4a7dae82c7ae2aa00496: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:19:49.087779 3814 master.cc:537] Master@127.28.235.252:39391 shutting down...
I20260926 07:19:49.088455 3804 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:19:49.090253 3230 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 -> Peer 0259221937c64b3188c7fdcf29a0efc7 (127.28.235.252:39391): Couldn't send request to peer 0259221937c64b3188c7fdcf29a0efc7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:19:49.095579 3814 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:49.096077 3814 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:49.096195 3814 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: stopping tablet replica
I20260926 07:19:49.096841 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.087357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.28.235.253:36465)
0926 07:19:49.087559 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.087564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.091180 (+ 3616us) server_negotiation.cc:407] Connection header received
0926 07:19:49.091372 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.091375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.091445 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.091510 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:49.092246 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.093087 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.095160 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.095834 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.095930 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.096067 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.096602 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.096702 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 07:19:49.097402 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.085461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36465 (local address 127.0.0.1:57216)
0926 07:19:49.091097 (+ 5636us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.091115 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.091254 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.091686 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.091689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.091708 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:49.092010 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.092017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.093738 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.093741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.095010 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.095022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.096313 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.096317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.096416 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.096480 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.097194 (+ 714us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.097282 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5552}
I20260926 07:19:49.098068 3852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 LEARNER]: Refusing update from remote peer 3335d929499c4a7dae82c7ae2aa00496: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:19:49.098631 3804 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 07:19:49.100857 3852 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0259221937c64b3188c7fdcf29a0efc7 (127.28.235.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.103255 3884 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3335d929499c4a7dae82c7ae2aa00496. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.103365 3884 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: This master's current role is: LEARNER
I20260926 07:19:49.103466 3884 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: NON_VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: true } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.103529 3884 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: This master's current role is: LEARNER
I20260926 07:19:49.104483 3804 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496: attempting to promote NON_VOTER ec96a06c50974233bbfdd1b858a3d28c to VOTER
I20260926 07:19:49.104983 3804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } }
I20260926 07:19:49.106927 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.106632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:39391 (local address 127.0.0.1:47844)
0926 07:19:49.106765 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.106844 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:39391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 07:19:49.107123 3852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 LEARNER]: Refusing update from remote peer 3335d929499c4a7dae82c7ae2aa00496: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 07:19:49.107774 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.107542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:39391 (local address 127.0.0.1:47846)
0926 07:19:49.107661 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.107713 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:39391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 07:19:49.107745 3803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 07:19:49.107998 3230 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 -> Peer 0259221937c64b3188c7fdcf29a0efc7 (127.28.235.252:39391): Couldn't send request to peer 0259221937c64b3188c7fdcf29a0efc7. Status: Network error: Client connection negotiation failed: client connection to 127.28.235.252:39391: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:19:49.110021 3805 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ec96a06c50974233bbfdd1b858a3d28c (127.28.235.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.110517 3852 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ec96a06c50974233bbfdd1b858a3d28c (127.28.235.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.111526 3805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.111621 3805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:49.111732 3805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.111794 3805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:49.112627 3991 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.112774 3991 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:19:49.116760 3814 master.cc:552] Master@127.28.235.252:39391 shutdown complete.
I20260926 07:19:49.116839 3814 master_runner.cc:305] Clearing existing system tablet
I20260926 07:19:49.117148 3814 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:49.117208 3814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:49.118624 3814 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:49.119557 3993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.119843 3814 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:49.119973 3814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:49.120015 3814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal
uuid: "0259221937c64b3188c7fdcf29a0efc7"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "281d1314bfd7cd3bde0009610ea27492"
server_key_iv: "c06a201ecefb52edfbac0443b7ddf613"
server_key_version: "encryptionkey@0"
I20260926 07:19:49.122210 3814 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:49.124539 3814 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:19:49.124631 3814 log.cc:1196] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 07:19:49.124869 3814 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Deleting consensus metadata
I20260926 07:19:49.125065 3814 master_runner.cc:315] Copying system tablet from 127.28.235.254:34501
I20260926 07:19:49.126281 3814 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: tablet copy: Beginning tablet copy session from remote peer at address 127.28.235.254:34501
I20260926 07:19:49.137950 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.126867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42122)
0926 07:19:49.129214 (+ 2347us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.129231 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.129367 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.129699 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.129702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.129932 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.130048 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.130056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.131198 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.131202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.132194 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.132205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.132821 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.132826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.132956 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.133595 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.133611 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.134064 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.136145 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.136149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.136151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.136436 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.136741 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.136746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.136750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.136868 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.137325 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.137328 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.137453 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.137578 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.137736 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2263,"thread_start_us":98,"threads_started":1}
I20260926 07:19:49.137967 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.126961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.28.235.254:34501)
0926 07:19:49.127093 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.127097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.129333 (+ 2236us) server_negotiation.cc:407] Connection header received
0926 07:19:49.129487 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.129490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.129548 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.129663 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.130206 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.131030 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.132333 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.132674 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.132780 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.134221 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.134224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.134227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.134256 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.136017 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.136543 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.136546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.136548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.136633 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.136976 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.136979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.136981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.137220 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.137313 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.137676 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.137795 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:19:49.139207 3263 tablet_copy_service.cc:140] P 3335d929499c4a7dae82c7ae2aa00496: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0259221937c64b3188c7fdcf29a0efc7 ({username='oryx', principal='oryx/127.28.235.252@KRBTEST.COM'} at 127.0.0.1:42122)
I20260926 07:19:49.139338 3263 tablet_copy_service.cc:161] P 3335d929499c4a7dae82c7ae2aa00496: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0259221937c64b3188c7fdcf29a0efc7 at {username='oryx', principal='oryx/127.28.235.252@KRBTEST.COM'} at 127.0.0.1:42122: session id = 0259221937c64b3188c7fdcf29a0efc7-00000000000000000000000000000000
I20260926 07:19:49.141170 3263 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:19:49.142518 3814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:19:49.144877 3814 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: tablet copy: Starting download of 0 data blocks...
I20260926 07:19:49.145097 3814 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: tablet copy: Starting download of 1 WAL segments...
I20260926 07:19:49.146776 3814 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:19:49.149305 3263 tablet_copy_service.cc:342] P 3335d929499c4a7dae82c7ae2aa00496: Request end of tablet copy session 0259221937c64b3188c7fdcf29a0efc7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.235.252@KRBTEST.COM'} at 127.0.0.1:42122
I20260926 07:19:49.149405 3263 tablet_copy_service.cc:433] P 3335d929499c4a7dae82c7ae2aa00496: ending tablet copy session 0259221937c64b3188c7fdcf29a0efc7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0259221937c64b3188c7fdcf29a0efc7
I20260926 07:19:49.158933 3814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:49.161137 4000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:49.161145 4001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:49.161489 3814 server_base.cc:1034] running on GCE node
W20260926 07:19:49.161151 4003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:49.163647 3814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:49.164064 3814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:49.165213 3814 hybrid_clock.cc:648] HybridClock initialized: now 1790407189165204 us; error 38 us; skew 500 ppm
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:49.167196 3814 init.cc:377] Logged in from keytab as oryx/127.28.235.252@KRBTEST.COM (short username oryx)
I20260926 07:19:49.167742 3814 webserver.cc:460] Webserver started at http://127.28.235.252:44693/ using document root <none> and password file <none>
I20260926 07:19:49.167948 3814 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:49.167989 3814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:49.169540 3814 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:19:49.170781 4010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.171120 3814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:49.171208 3814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal
uuid: "0259221937c64b3188c7fdcf29a0efc7"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "281d1314bfd7cd3bde0009610ea27492"
server_key_iv: "c06a201ecefb52edfbac0443b7ddf613"
server_key_version: "encryptionkey@0"
I20260926 07:19:49.171298 3814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:49.188884 3568 heartbeater.cc:503] Master 127.28.235.254:34501 was elected leader, sending a full tablet report...
I20260926 07:19:49.191501 3814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260926 07:19:49.193288 3989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:49.193578 3989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:49.193662 3989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:19:49.193720 3814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:49.198997 3814 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.252:39391
I20260926 07:19:49.199023 4065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.252:39391 every 8 connection(s)
W20260926 07:19:49.199347 3989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:49.199421 3989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:49.199522 3989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:49.199644 3814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
I20260926 07:19:49.202113 4066 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:19:49.203157 4066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Bootstrap starting.
I20260926 07:19:49.207429 3989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab
--principal=oryx/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:35801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=35725
--tserver_master_addrs=127.28.235.254:34501,127.28.235.253:36465,127.28.235.252:39391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:49.208822 3989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:49.210222 3989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:19:49.215022 4066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:19:49.215405 4066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Bootstrap complete.
I20260926 07:19:49.215978 4066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } }
I20260926 07:19:49.216140 4066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0259221937c64b3188c7fdcf29a0efc7, State: Initialized, Role: LEARNER
I20260926 07:19:49.216311 4066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } }
I20260926 07:19:49.216687 4069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.216792 4066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:49.216895 4069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: This master's current role is: LEARNER
I20260926 07:19:49.220418 4085 catalog_manager.cc:1259] Loaded cluster ID: 2d2fff5f73d349f7a0f0a40ca3672ad4
I20260926 07:19:49.220468 4085 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: loading cluster ID for follower catalog manager: success
I20260926 07:19:49.222699 4085 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: acquiring CA information for follower catalog manager: success
W20260926 07:19:49.223143 4075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:49.223311 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:49.223112 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:49.224258 4085 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:19:49.224578 3989 server_base.cc:1034] running on GCE node
I20260926 07:19:49.225117 3989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:49.225773 3989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:49.226933 3989 hybrid_clock.cc:648] HybridClock initialized: now 1790407189226919 us; error 28 us; skew 500 ppm
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:49.230050 3989 init.cc:377] Logged in from keytab as oryx/127.28.235.193@KRBTEST.COM (short username oryx)
I20260926 07:19:49.231006 3989 webserver.cc:460] Webserver started at http://127.28.235.193:35725/ using document root <none> and password file <none>
I20260926 07:19:49.231627 3989 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:49.231735 3989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:49.235972 3989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:49.238263 4092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.239621 3989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:19:49.239745 3989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "76925f7ba8f842d1a98627db55b76657"
format_stamp: "Formatted at 2026-09-26 07:19:47 on dist-test-slave-7zlw"
server_key: "2c25a7f9bab9f15cf5c570c931f71efa"
server_key_iv: "5d8d23f914ebaf543e17eee06c678b70"
server_key_version: "encryptionkey@0"
I20260926 07:19:49.240283 3989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:49.262651 3989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:49.266328 3989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:49.266597 3989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:49.267225 3989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:49.268102 3989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:49.268147 3989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.268218 3989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:49.268249 3989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.277930 3989 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:35801
I20260926 07:19:49.277956 4205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:35801 every 8 connection(s)
I20260926 07:19:49.279235 3989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:49.282382 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 3989
I20260926 07:19:49.282619 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 3439
I20260926 07:19:49.293319 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:41571
--local_ip_for_outbound_sockets=127.28.235.194
--tserver_master_addrs=127.28.235.254:34501
--webserver_port=36839
--webserver_interface=127.28.235.194
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab
--principal=oryx/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.235.254:34501,127.28.235.253:36465
--tserver_master_addrs=127.28.235.254:34501,127.28.235.253:36465,127.28.235.252:39391 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.252@KRBTEST.COM
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.253@KRBTEST.COM
I20260926 07:19:49.300565 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.281284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:44407 (local address 127.28.235.254:34501)
0926 07:19:49.281406 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.281417 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.283055 (+ 1638us) server_negotiation.cc:407] Connection header received
0926 07:19:49.284622 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.284629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.284702 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.284766 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.286955 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.287803 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.290181 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.290556 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.290652 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.295772 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.295778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.295786 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.295816 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.298571 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.299120 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.299124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.299126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.299209 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.299564 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.299567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.299569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.299787 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.299879 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.300246 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.300383 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 07:19:49.300827 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.283272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:51721 (local address 127.28.235.253:36465)
0926 07:19:49.283426 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.283432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.285545 (+ 2113us) server_negotiation.cc:407] Connection header received
0926 07:19:49.285743 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.285750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.285818 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.285879 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.287020 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.288034 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.290242 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.290773 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.290878 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.296118 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.296122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.296127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.296158 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.298280 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.299082 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.299089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.299093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.299201 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.299626 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.299631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.299635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.299928 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.300037 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.300476 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.300633 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:19:49.301676 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.281764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:39391 (local address 127.28.235.193:48553)
0926 07:19:49.283528 (+ 1764us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.283541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.284371 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.285006 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.285054 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.285751 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.286767 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.286781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.288167 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.288170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.290097 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.290108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.290836 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.290839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.291175 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.292214 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.292232 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.295105 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.297648 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.297661 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.297682 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.298094 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.298468 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.298472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.298475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.298840 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.299420 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.299427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.299818 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.300033 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.300410 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1653,"mutex_wait_us":708,"thread_start_us":132,"threads_started":1}
I20260926 07:19:49.301977 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.281764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.193:44407)
0926 07:19:49.282875 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.282938 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.284357 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.285035 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.285044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.285751 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.286760 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.286781 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.287997 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.288000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.290002 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.290013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.290836 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.290839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.291180 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.292080 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.292097 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.295619 (+ 3522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.298704 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.298708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.298710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.298998 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.299336 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.299339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.299341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.299448 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.299940 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.299942 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.300058 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.300799 (+ 741us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.300955 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":714,"spinlock_wait_cycles":12032,"thread_start_us":157,"threads_started":1}
I20260926 07:19:49.301838 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.282915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36465 (local address 127.28.235.193:51721)
0926 07:19:49.285436 (+ 2521us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.285466 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.285620 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.286048 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.286055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.286571 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.286855 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.286868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.288195 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.288202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.290085 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.290100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.291086 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.291093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.291274 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.292033 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.292062 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.295601 (+ 3539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.298417 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.298422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.298427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.298860 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.299343 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.299349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.299354 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.299493 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.300113 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.300119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.300241 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.301083 (+ 842us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.301508 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2430,"mutex_wait_us":418,"thread_start_us":78,"threads_started":1}
I20260926 07:19:49.302500 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.282851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:48553 (local address 127.28.235.252:39391)
0926 07:19:49.283225 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.283230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.283626 (+ 396us) server_negotiation.cc:407] Connection header received
0926 07:19:49.284608 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.284612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.284678 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.284797 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.286966 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.288036 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.290234 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.290581 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.290671 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.295325 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.295330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.295333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.295367 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.297496 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.298231 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.298235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.298238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.298335 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.298969 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.298973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.298975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.299264 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.299364 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.302163 (+ 2799us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.302313 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":86,"threads_started":1}
I20260926 07:19:49.303131 4207 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34501
I20260926 07:19:49.303136 4208 heartbeater.cc:347] Connected to a master server at 127.28.235.253:36465
I20260926 07:19:49.303239 4206 heartbeater.cc:347] Connected to a master server at 127.28.235.252:39391
I20260926 07:19:49.303422 4206 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:49.303421 4208 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:49.303421 4207 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:49.304129 4208 heartbeater.cc:511] Master 127.28.235.253:36465 requested a full tablet report, sending...
I20260926 07:19:49.304129 4207 heartbeater.cc:511] Master 127.28.235.254:34501 requested a full tablet report, sending...
I20260926 07:19:49.304129 4206 heartbeater.cc:511] Master 127.28.235.252:39391 requested a full tablet report, sending...
I20260926 07:19:49.305517 3236 ts_manager.cc:194] Re-registered known tserver with Master: 76925f7ba8f842d1a98627db55b76657 (127.28.235.193:35801)
I20260926 07:19:49.306075 3841 ts_manager.cc:194] Registered new tserver with Master: 76925f7ba8f842d1a98627db55b76657 (127.28.235.193:35801)
I20260926 07:19:49.306190 3236 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:44407
I20260926 07:19:49.307966 4030 ts_manager.cc:194] Registered new tserver with Master: 76925f7ba8f842d1a98627db55b76657 (127.28.235.193:35801)
I20260926 07:19:49.409839 3702 heartbeater.cc:503] Master 127.28.235.254:34501 was elected leader, sending a full tablet report...
W20260926 07:19:49.417037 4214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:49.417379 4214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:49.417480 4214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:49.421856 4214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:49.421936 4214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:49.422034 4214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:49.427579 4214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab
--principal=oryx/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:41571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=36839
--tserver_master_addrs=127.28.235.254:34501,127.28.235.253:36465,127.28.235.252:39391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:49.428771 4214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:49.429931 4214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:49.439990 4220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:49.439972 4219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:49.440308 4214 server_base.cc:1034] running on GCE node
W20260926 07:19:49.440418 4222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:49.440846 4214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:49.441612 4214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:49.442814 4214 hybrid_clock.cc:648] HybridClock initialized: now 1790407189442792 us; error 34 us; skew 500 ppm
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:49.445807 4214 init.cc:377] Logged in from keytab as oryx/127.28.235.194@KRBTEST.COM (short username oryx)
I20260926 07:19:49.446663 4214 webserver.cc:460] Webserver started at http://127.28.235.194:36839/ using document root <none> and password file <none>
I20260926 07:19:49.447211 4214 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:49.447284 4214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:49.451442 4214 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:19:49.453770 4229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.455329 4214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:49.455528 4214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "89e80c04d6c24d0983e9a0d4f71ae59f"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "f92ad4c0811eb0ffa65f1a7f05200b84"
server_key_iv: "d78291bb3cccf89da5acebef4bca8a93"
server_key_version: "encryptionkey@0"
I20260926 07:19:49.456152 4214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:49.479269 4214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:49.483422 4214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:49.483651 4214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:49.484318 4214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:49.485352 4214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:49.485428 4214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.485507 4214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:49.485562 4214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.495368 4214 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:41571
I20260926 07:19:49.495437 4342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:41571 every 8 connection(s)
I20260926 07:19:49.496454 4214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:49.501696 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 4214
I20260926 07:19:49.501871 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 3573
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.253@KRBTEST.COM
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.252@KRBTEST.COM
I20260926 07:19:49.511576 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:45051
--local_ip_for_outbound_sockets=127.28.235.195
--tserver_master_addrs=127.28.235.254:34501
--webserver_port=36481
--webserver_interface=127.28.235.195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab
--principal=oryx/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.235.254:34501,127.28.235.253:36465
--tserver_master_addrs=127.28.235.254:34501,127.28.235.253:36465,127.28.235.252:39391 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:19:49.516844 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.498405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:38471 (local address 127.28.235.254:34501)
0926 07:19:49.498543 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.498548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.499929 (+ 1381us) server_negotiation.cc:407] Connection header received
0926 07:19:49.500926 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.500930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.500995 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.501084 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.503065 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.503925 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.506215 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.506599 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.506678 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.511420 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.511426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.511429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.511463 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.513983 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.514917 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.514924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.514925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.515025 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.515459 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.515463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.515465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.515695 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.515773 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.516526 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.516660 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:19:49.516844 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.498371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:52373 (local address 127.28.235.252:39391)
0926 07:19:49.498512 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.498517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.499683 (+ 1166us) server_negotiation.cc:407] Connection header received
0926 07:19:49.500905 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.500908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.500980 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.501105 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.503085 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.503986 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.506200 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.506648 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.506741 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.511917 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.511922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.511924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.511957 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.514267 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.514991 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.514994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.514996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.515083 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.515472 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.515476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.515477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.515695 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.515773 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.516525 (+ 752us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.516660 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:19:49.516903 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.500053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:53973 (local address 127.28.235.253:36465)
0926 07:19:49.500181 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.500186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.500258 (+ 72us) server_negotiation.cc:407] Connection header received
0926 07:19:49.501056 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.501063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.501117 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.501174 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.502895 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.503829 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.506198 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.506588 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.506678 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.511838 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.511844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.511850 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.511882 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.514086 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.514855 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.514861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.514866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.514953 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.515362 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.515369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.515373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.515645 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.515728 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.516533 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.516681 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:19:49.518869 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.498836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.194:38471)
0926 07:19:49.499802 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.499816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.500721 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.501286 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.501290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.502379 (+ 1089us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.502935 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.502946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.504055 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.504061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.506041 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.506056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.506904 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.506908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.507366 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.508334 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.508347 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.511209 (+ 2862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.514142 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.514153 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.514175 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.514768 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.515209 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.515212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.515214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.515337 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.515857 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.515860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.516201 (+ 341us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.518016 (+ 1815us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.518164 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":886,"spinlock_wait_cycles":414720,"thread_start_us":153,"threads_started":1}
I20260926 07:19:49.518870 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.499910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36465 (local address 127.28.235.194:53973)
0926 07:19:49.500173 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.500188 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.500721 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.501325 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.501328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.502042 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.502736 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.502753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.504013 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.504017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.506068 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.506074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.506904 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.506908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.507300 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.508479 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.508500 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.511671 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.514226 (+ 2555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.514232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.514236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.514738 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.515063 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.515069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.515074 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.515251 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.515815 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.515824 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.516202 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.517175 (+ 973us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.517649 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":725,"spinlock_wait_cycles":30720,"thread_start_us":85,"threads_started":1}
I20260926 07:19:49.518869 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.498836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:39391 (local address 127.28.235.194:52373)
0926 07:19:49.499495 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.499564 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.500696 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.501281 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.501288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.502539 (+ 1251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.502736 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.502755 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.504137 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.504143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.506060 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.506075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.506916 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.506922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.507336 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.508293 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.508317 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.511743 (+ 3426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.514406 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.514411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.514414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.514875 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.515209 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.515212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.515214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.515345 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.515857 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.515860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.516201 (+ 341us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.517124 (+ 923us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.517751 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":625,"spinlock_wait_cycles":40192,"thread_start_us":698,"threads_started":1}
I20260926 07:19:49.520248 4344 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34501
I20260926 07:19:49.520362 4345 heartbeater.cc:347] Connected to a master server at 127.28.235.253:36465
I20260926 07:19:49.520522 4344 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:49.520524 4345 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:49.521041 4343 heartbeater.cc:347] Connected to a master server at 127.28.235.252:39391
I20260926 07:19:49.521121 4343 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:49.521222 4345 heartbeater.cc:511] Master 127.28.235.253:36465 requested a full tablet report, sending...
I20260926 07:19:49.521222 4344 heartbeater.cc:511] Master 127.28.235.254:34501 requested a full tablet report, sending...
I20260926 07:19:49.521322 4343 heartbeater.cc:511] Master 127.28.235.252:39391 requested a full tablet report, sending...
I20260926 07:19:49.522527 3236 ts_manager.cc:194] Re-registered known tserver with Master: 89e80c04d6c24d0983e9a0d4f71ae59f (127.28.235.194:41571)
I20260926 07:19:49.522676 4030 ts_manager.cc:194] Registered new tserver with Master: 89e80c04d6c24d0983e9a0d4f71ae59f (127.28.235.194:41571)
I20260926 07:19:49.523207 3841 ts_manager.cc:194] Registered new tserver with Master: 89e80c04d6c24d0983e9a0d4f71ae59f (127.28.235.194:41571)
I20260926 07:19:49.523417 3236 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:38471
W20260926 07:19:49.629220 4350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:49.629484 4350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:49.629528 4350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:19:49.631168 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.627521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.28.235.252:39391)
0926 07:19:49.627668 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.627672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.627782 (+ 110us) server_negotiation.cc:407] Connection header received
0926 07:19:49.627925 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.627927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.627981 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.628160 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:49.628435 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.628954 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.630156 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.630623 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.630758 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.630914 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.630954 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.631031 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:19:49.631172 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.627386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:39391 (local address 127.0.0.1:47856)
0926 07:19:49.627682 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.627698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.627824 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.628172 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.628176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.628195 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:49.628314 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.628320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.629123 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.629128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.630020 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.630031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.630755 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.630758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.630850 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.630875 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.630942 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.631028 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260926 07:19:49.631784 4040 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 07:19:49.632185 4069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3335d929499c4a7dae82c7ae2aa00496. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.632324 4069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: This master's current role is: LEARNER
W20260926 07:19:49.634558 4350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:49.634673 4350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:49.634789 4350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:49.640996 4350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab
--principal=oryx/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:45051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=36481
--tserver_master_addrs=127.28.235.254:34501,127.28.235.253:36465,127.28.235.252:39391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:49.642283 4350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:49.643402 4350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:49.653602 4356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:49.653620 4359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:49.653815 4357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:49.653842 4350 server_base.cc:1034] running on GCE node
I20260926 07:19:49.654381 4350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:49.654986 4350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:49.656178 4350 hybrid_clock.cc:648] HybridClock initialized: now 1790407189656141 us; error 46 us; skew 500 ppm
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:49.659327 4350 init.cc:377] Logged in from keytab as oryx/127.28.235.195@KRBTEST.COM (short username oryx)
I20260926 07:19:49.660328 4350 webserver.cc:460] Webserver started at http://127.28.235.195:36481/ using document root <none> and password file <none>
I20260926 07:19:49.660957 4350 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:49.661095 4350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:49.665248 4350 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:49.667953 4366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.669728 4350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:49.669847 4350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "ab96d3b6385040429868d9fc330d6b92"
format_stamp: "Formatted at 2026-09-26 07:19:48 on dist-test-slave-7zlw"
server_key: "33055e72b90e27d1a6c6334a117962ac"
server_key_iv: "e42b929f528a1becd1d176cc8a4db163"
server_key_version: "encryptionkey@0"
I20260926 07:19:49.670456 4350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:49.714046 4350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:49.717718 4350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:49.717914 4350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:49.718626 4350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:19:49.719614 4350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:49.719664 4350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.719707 4350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:49.719756 4350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:49.729076 4350 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:45051
I20260926 07:19:49.729116 4479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:45051 every 8 connection(s)
I20260926 07:19:49.730242 4350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:49.740044 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 4350
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.252@KRBTEST.COM
Sep 26 07:19:49 dist-test-slave-7zlw krb5kdc[3198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407189, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.253@KRBTEST.COM
I20260926 07:19:49.747370 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.732179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:47657 (local address 127.28.235.253:36465)
0926 07:19:49.732325 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.732329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.733406 (+ 1077us) server_negotiation.cc:407] Connection header received
0926 07:19:49.734560 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.734564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.734623 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.734731 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.736003 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.736530 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.737804 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.738049 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.738110 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.743188 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.743195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.743199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.743227 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.745055 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.745563 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.745567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.745569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.745653 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.746059 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.746064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.746068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.746299 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.746385 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.747019 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.747175 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:19:49.747494 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.732226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:43663 (local address 127.28.235.254:34501)
0926 07:19:49.732351 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.732355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.733690 (+ 1335us) server_negotiation.cc:407] Connection header received
0926 07:19:49.734570 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.734573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.734628 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.734682 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.736025 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.736805 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.738232 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.738456 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.738514 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.742048 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.742052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.742054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.742085 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.744725 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.746090 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.746094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.746097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.746178 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.746491 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.746494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.746496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.746726 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.746825 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.747231 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.747352 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:19:49.748512 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.733707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:39391 (local address 127.28.235.195:38175)
0926 07:19:49.733999 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.734019 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.734358 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.734907 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.734915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.735390 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.735856 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.735870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.736947 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.736950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.738095 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.738105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.738690 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.738693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.739373 (+ 680us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.739850 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.739868 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.741524 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.746044 (+ 4520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.746048 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.746050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.746293 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.746672 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.746675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.746677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.746760 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.747259 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.747262 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.747344 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.747459 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.747605 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":700,"spinlock_wait_cycles":18304,"thread_start_us":100,"threads_started":1}
I20260926 07:19:49.748782 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.732609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.195:43663)
0926 07:19:49.733583 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.733594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.734358 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.734907 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.734915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.735390 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.735856 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.735870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.736947 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.736950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.738095 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.738105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.738678 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.738683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.738860 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.739354 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.739370 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.741877 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.744868 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.744877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.744899 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.745328 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.746281 (+ 953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.746284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.746286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.746374 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.746835 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.746838 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.746917 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.747012 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.747992 (+ 980us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":1025,"spinlock_wait_cycles":74112,"thread_start_us":127,"threads_started":1}
I20260926 07:19:49.749202 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.733908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:38175 (local address 127.28.235.252:39391)
0926 07:19:49.734043 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.734049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.734067 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:19:49.734571 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.734574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.734629 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.734682 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.736012 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.736806 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.738221 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.738548 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.738658 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.741707 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.741711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.741714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.741743 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.745934 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.746466 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.746473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.746475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.746566 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.746862 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.746865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.746867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.747123 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.747216 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.748888 (+ 1672us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.749063 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:19:49.749661 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.732609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36465 (local address 127.28.235.195:47657)
0926 07:19:49.733202 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.733267 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.734336 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.734907 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.734913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.735329 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:49.735853 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.735866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.736668 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.736673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.737628 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.737637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.738319 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.738322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.738518 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.739209 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:49.739226 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:49.743018 (+ 3792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:49.745169 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.745172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.745177 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.745439 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.745760 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:49.745764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:49.745766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:49.745950 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:49.746437 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:49.746446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:49.746712 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.746940 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.747333 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":4,"thread_start_us":734,"threads_started":1}
I20260926 07:19:49.749938 4481 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34501
I20260926 07:19:49.750027 4480 heartbeater.cc:347] Connected to a master server at 127.28.235.252:39391
I20260926 07:19:49.751272 4481 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:49.751905 4481 heartbeater.cc:511] Master 127.28.235.254:34501 requested a full tablet report, sending...
I20260926 07:19:49.750977 4482 heartbeater.cc:347] Connected to a master server at 127.28.235.253:36465
I20260926 07:19:49.752537 4482 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:49.752722 4482 heartbeater.cc:511] Master 127.28.235.253:36465 requested a full tablet report, sending...
I20260926 07:19:49.751272 4480 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:49.753106 4480 heartbeater.cc:511] Master 127.28.235.252:39391 requested a full tablet report, sending...
I20260926 07:19:49.754240 4030 ts_manager.cc:194] Registered new tserver with Master: ab96d3b6385040429868d9fc330d6b92 (127.28.235.195:45051)
I20260926 07:19:49.754315 3841 ts_manager.cc:194] Registered new tserver with Master: ab96d3b6385040429868d9fc330d6b92 (127.28.235.195:45051)
I20260926 07:19:49.754974 3236 ts_manager.cc:194] Re-registered known tserver with Master: ab96d3b6385040429868d9fc330d6b92 (127.28.235.195:45051)
I20260926 07:19:49.755803 3236 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:43663
I20260926 07:19:49.755882 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.742847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.28.235.252:39391)
0926 07:19:49.743148 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.743153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.743723 (+ 570us) server_negotiation.cc:407] Connection header received
0926 07:19:49.743895 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.743898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.743949 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.744014 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.745229 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.746041 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.749455 (+ 3414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.749839 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.749917 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.752206 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.752218 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.752221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.752241 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.754067 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.754565 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.754568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.754570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.754654 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.754929 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.754934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.754939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.755160 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.755261 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.755597 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.755718 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":115,"threads_started":1}
I20260926 07:19:49.758826 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.742101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.28.235.253:36465)
0926 07:19:49.743433 (+ 1332us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.743437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.743450 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:19:49.743579 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.743582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.743688 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.743772 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.745229 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.749846 (+ 4617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.751139 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.751455 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.751551 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.754161 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.754164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.754167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.754191 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.756449 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.757211 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.757214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.757216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.757307 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.757614 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.757617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.757619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.757822 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.757942 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.758471 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.758605 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":82,"threads_started":1}
I20260926 07:19:49.761161 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.744191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.28.235.254:34501)
0926 07:19:49.747658 (+ 3467us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.747662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.752027 (+ 4365us) server_negotiation.cc:407] Connection header received
0926 07:19:49.752188 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.752191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.752237 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.752318 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.753164 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.754908 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.756033 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.756414 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.756502 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.757907 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:49.757910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:49.757913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:49.757940 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:49.759693 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.760095 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.760097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.760098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.760175 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:49.760429 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:49.760431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:49.760432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:49.760605 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:49.760737 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.760876 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.760964 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3409,"thread_start_us":70,"threads_started":1}
I20260926 07:19:49.763841 3236 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:19:49.767367 3236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:49.770255 3804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: NON_VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: true } } }
I20260926 07:19:49.770431 3804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:49.773932 3804 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496: attempting to promote NON_VOTER 0259221937c64b3188c7fdcf29a0efc7 to VOTER
I20260926 07:19:49.774262 3804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } }
I20260926 07:19:49.776353 4039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 LEARNER]: Refusing update from remote peer 3335d929499c4a7dae82c7ae2aa00496: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 07:19:49.776461 3852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Refusing update from remote peer 3335d929499c4a7dae82c7ae2aa00496: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 07:19:49.776890 4498 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 07:19:49.777447 4498 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 07:19:49.779124 4500 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0259221937c64b3188c7fdcf29a0efc7 (127.28.235.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:49.780465 4500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:49.780576 4500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:49.780604 4499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:49.780632 3804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:49.780737 4499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:49.780745 3804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:49.781294 3852 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0259221937c64b3188c7fdcf29a0efc7 (127.28.235.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:49.781297 4039 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0259221937c64b3188c7fdcf29a0efc7 (127.28.235.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:49.783186 4069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:49.783299 4069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:19:49.783457 4508 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3335d929499c4a7dae82c7ae2aa00496" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:49.783581 4508 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:19:49.791818 4354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.785889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:45051 (local address 127.0.0.1:48580)
0926 07:19:49.786015 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.786032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.786118 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.787347 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.787350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.787372 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:49.787478 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.787488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.788925 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.788927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.790104 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.790114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.791411 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.791415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.791515 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.791539 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.791617 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.791697 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 07:19:49.791958 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.786087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.28.235.195:45051)
0926 07:19:49.786427 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.786441 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.786464 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:19:49.786566 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.786574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.786828 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.787036 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:49.787609 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.788806 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.790229 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.791097 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.791215 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.791346 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.791681 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.791788 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":120,"threads_started":1}
I20260926 07:19:49.793615 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.786580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:41571 (local address 127.0.0.1:59890)
0926 07:19:49.789210 (+ 2630us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.789224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.789313 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.789778 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.789782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.789800 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:49.789888 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.789893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.791141 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.791147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.792067 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.792080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.793197 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.793201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.793286 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.793311 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.793378 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.793457 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2562,"thread_start_us":113,"threads_started":1}
I20260926 07:19:49.793634 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.787081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.28.235.194:41571)
0926 07:19:49.787353 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.787364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.789325 (+ 1961us) server_negotiation.cc:407] Connection header received
0926 07:19:49.789414 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.789419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.789618 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.789779 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:49.790005 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.791045 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.792221 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.793062 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.793213 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.793355 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.793409 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.793498 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260926 07:19:49.795364 4414 tablet_service.cc:1467] Processing CreateTablet for tablet 164d37922bd54da9a897e28deed941ec (DEFAULT_TABLE table=test-table [id=831c383a53624c0a9038695d1c83ca82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:49.796078 4277 tablet_service.cc:1467] Processing CreateTablet for tablet 164d37922bd54da9a897e28deed941ec (DEFAULT_TABLE table=test-table [id=831c383a53624c0a9038695d1c83ca82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:49.796620 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164d37922bd54da9a897e28deed941ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:49.796869 4277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164d37922bd54da9a897e28deed941ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:49.799094 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.786175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:35801 (local address 127.0.0.1:45128)
0926 07:19:49.786632 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.786787 (+ 155us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.786871 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.787782 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.787785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.787800 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:49.787871 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.787875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.789207 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.789210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.790248 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.790256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.798631 (+ 8375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.798637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.798743 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.798771 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.798850 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.798944 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":115,"threads_started":1}
I20260926 07:19:49.799568 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.786617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.28.235.193:35801)
0926 07:19:49.787315 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.787333 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.787355 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:19:49.787434 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.787440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.787654 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.787815 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:49.787965 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.789078 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.790380 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.798235 (+ 7855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.798375 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.798552 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.799300 (+ 748us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.799420 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":84,"threads_started":1}
I20260926 07:19:49.802819 4140 tablet_service.cc:1467] Processing CreateTablet for tablet 164d37922bd54da9a897e28deed941ec (DEFAULT_TABLE table=test-table [id=831c383a53624c0a9038695d1c83ca82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:49.803956 4140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164d37922bd54da9a897e28deed941ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:49.805786 4514 tablet_bootstrap.cc:492] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: Bootstrap starting.
I20260926 07:19:49.806921 4515 tablet_bootstrap.cc:492] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: Bootstrap starting.
I20260926 07:19:49.808907 4515 tablet_bootstrap.cc:654] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:49.809681 4515 log.cc:824] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:49.811717 4514 tablet_bootstrap.cc:654] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:49.811970 4515 tablet_bootstrap.cc:492] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: No bootstrap required, opened a new log
I20260926 07:19:49.812187 4515 ts_tablet_manager.cc:1397] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260926 07:19:49.812647 4514 log.cc:824] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:49.813711 4517 tablet_bootstrap.cc:492] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: Bootstrap starting.
I20260926 07:19:49.814721 4514 tablet_bootstrap.cc:492] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: No bootstrap required, opened a new log
I20260926 07:19:49.814961 4514 ts_tablet_manager.cc:1397] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20260926 07:19:49.816090 4517 tablet_bootstrap.cc:654] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:49.816963 4517 log.cc:824] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:49.817126 4515 raft_consensus.cc:348] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } } peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } }
I20260926 07:19:49.817420 4515 raft_consensus.cc:374] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:49.817484 4515 raft_consensus.cc:729] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab96d3b6385040429868d9fc330d6b92, State: Initialized, Role: FOLLOWER
I20260926 07:19:49.818066 4515 consensus_queue.cc:260] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } } peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } }
I20260926 07:19:49.819195 4515 ts_tablet_manager.cc:1428] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:19:49.819368 4481 heartbeater.cc:503] Master 127.28.235.254:34501 was elected leader, sending a full tablet report...
I20260926 07:19:49.820645 4517 tablet_bootstrap.cc:492] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: No bootstrap required, opened a new log
I20260926 07:19:49.820864 4517 ts_tablet_manager.cc:1397] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:19:49.822084 4514 raft_consensus.cc:348] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } } peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } }
I20260926 07:19:49.822376 4514 raft_consensus.cc:374] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:49.822428 4514 raft_consensus.cc:729] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e80c04d6c24d0983e9a0d4f71ae59f, State: Initialized, Role: FOLLOWER
I20260926 07:19:49.822953 4514 consensus_queue.cc:260] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } } peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } }
I20260926 07:19:49.824014 4514 ts_tablet_manager.cc:1428] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20260926 07:19:49.824420 4344 heartbeater.cc:503] Master 127.28.235.254:34501 was elected leader, sending a full tablet report...
I20260926 07:19:49.825034 4517 raft_consensus.cc:348] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } } peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } }
I20260926 07:19:49.825302 4517 raft_consensus.cc:374] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:49.825352 4517 raft_consensus.cc:729] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76925f7ba8f842d1a98627db55b76657, State: Initialized, Role: FOLLOWER
I20260926 07:19:49.825827 4517 consensus_queue.cc:260] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } } peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } }
I20260926 07:19:49.827072 4517 ts_tablet_manager.cc:1428] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:19:49.828135 4207 heartbeater.cc:503] Master 127.28.235.254:34501 was elected leader, sending a full tablet report...
I20260926 07:19:49.906214 4520 raft_consensus.cc:482] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:49.906489 4520 raft_consensus.cc:504] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } } peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } }
I20260926 07:19:49.907897 4520 leader_election.cc:288] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76925f7ba8f842d1a98627db55b76657 (127.28.235.193:35801), 89e80c04d6c24d0983e9a0d4f71ae59f (127.28.235.194:41571)
I20260926 07:19:49.912492 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.908296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:41571 (local address 127.28.235.195:53739)
0926 07:19:49.908665 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.908686 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.908819 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.909146 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.909149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.909177 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:49.909302 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.909308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.910144 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.910149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.911202 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.911213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.912036 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.912039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.912166 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.912193 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.912272 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.912336 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":158,"spinlock_wait_cycles":768}
I20260926 07:19:49.912526 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.908403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:53739 (local address 127.28.235.194:41571)
0926 07:19:49.908533 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.908538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.908776 (+ 238us) server_negotiation.cc:407] Connection header received
0926 07:19:49.908955 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.908960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.909030 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.909134 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:49.909420 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.910015 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.911308 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.911883 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.911996 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.912038 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.912313 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.912404 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:19:49.912526 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.908296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:35801 (local address 127.28.235.195:52779)
0926 07:19:49.908647 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:49.908659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:49.908788 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:49.909109 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:49.909112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:49.909127 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:49.909249 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.909263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.910013 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.910019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.911111 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:49.911122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.912102 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.912111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:49.912186 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.912203 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 07:19:49.912295 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:19:49.912370 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":91}
I20260926 07:19:49.912825 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.908392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:52779 (local address 127.28.235.193:35801)
0926 07:19:49.908538 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.908544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.908735 (+ 191us) server_negotiation.cc:407] Connection header received
0926 07:19:49.908910 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.908913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.908967 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.909096 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:49.909367 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.909893 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.911241 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.911855 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.911982 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.912038 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.912611 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.912704 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:19:49.913305 4297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164d37922bd54da9a897e28deed941ec" candidate_uuid: "ab96d3b6385040429868d9fc330d6b92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" is_pre_election: true
I20260926 07:19:49.913630 4297 raft_consensus.cc:2454] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab96d3b6385040429868d9fc330d6b92 in term 0.
I20260926 07:19:49.913676 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164d37922bd54da9a897e28deed941ec" candidate_uuid: "ab96d3b6385040429868d9fc330d6b92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76925f7ba8f842d1a98627db55b76657" is_pre_election: true
I20260926 07:19:49.913995 4160 raft_consensus.cc:2454] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab96d3b6385040429868d9fc330d6b92 in term 0.
I20260926 07:19:49.914209 4368 leader_election.cc:302] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e80c04d6c24d0983e9a0d4f71ae59f, ab96d3b6385040429868d9fc330d6b92; no voters:
I20260926 07:19:49.914565 4520 raft_consensus.cc:2781] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:49.914661 4520 raft_consensus.cc:482] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:49.914724 4520 raft_consensus.cc:3037] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:49.915906 4520 raft_consensus.cc:504] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } } peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } }
I20260926 07:19:49.916337 4520 leader_election.cc:288] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [CANDIDATE]: Term 1 election: Requested vote from peers 76925f7ba8f842d1a98627db55b76657 (127.28.235.193:35801), 89e80c04d6c24d0983e9a0d4f71ae59f (127.28.235.194:41571)
I20260926 07:19:49.916729 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164d37922bd54da9a897e28deed941ec" candidate_uuid: "ab96d3b6385040429868d9fc330d6b92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76925f7ba8f842d1a98627db55b76657"
I20260926 07:19:49.916720 4297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164d37922bd54da9a897e28deed941ec" candidate_uuid: "ab96d3b6385040429868d9fc330d6b92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f"
I20260926 07:19:49.916893 4297 raft_consensus.cc:3037] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:49.916887 4160 raft_consensus.cc:3037] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:49.918207 4297 raft_consensus.cc:2454] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab96d3b6385040429868d9fc330d6b92 in term 1.
I20260926 07:19:49.918236 4160 raft_consensus.cc:2454] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab96d3b6385040429868d9fc330d6b92 in term 1.
I20260926 07:19:49.918519 4368 leader_election.cc:302] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e80c04d6c24d0983e9a0d4f71ae59f, ab96d3b6385040429868d9fc330d6b92; no voters:
I20260926 07:19:49.918728 4520 raft_consensus.cc:2781] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:49.919031 4520 raft_consensus.cc:686] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 1 LEADER]: Becoming Leader. State: Replica: ab96d3b6385040429868d9fc330d6b92, State: Running, Role: LEADER
I20260926 07:19:49.919410 4520 consensus_queue.cc:237] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } } peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } }
I20260926 07:19:49.923192 3243 catalog_manager.cc:5495] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab96d3b6385040429868d9fc330d6b92 (127.28.235.195). New cstate: current_term: 1 leader_uuid: "ab96d3b6385040429868d9fc330d6b92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:49.950393 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.946683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.28.235.195:45051)
0926 07:19:49.946827 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.946832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.946847 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:19:49.947072 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.947075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.947133 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.947248 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:49.947485 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.948011 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.948868 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:49.949106 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:49.949219 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:49.949532 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:49.949967 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:49.950103 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:49.950165 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 07:19:49.950219 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:19:49.956598 4160 raft_consensus.cc:1261] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657 [term 1 FOLLOWER]: Refusing update from remote peer ab96d3b6385040429868d9fc330d6b92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:49.956600 4297 raft_consensus.cc:1261] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f [term 1 FOLLOWER]: Refusing update from remote peer ab96d3b6385040429868d9fc330d6b92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:49.957338 4523 consensus_queue.cc:1035] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:49.957528 4520 consensus_queue.cc:1035] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:49.967773 4527 mvcc.cc:205] Tried to move back new op lower bound from 7333507850054914048 to 7333507849912950784. Current Snapshot: MvccSnapshot[applied={T|T < 7333507850054914048}]
I20260926 07:19:49.971657 4528 mvcc.cc:205] Tried to move back new op lower bound from 7333507850054914048 to 7333507849912950784. Current Snapshot: MvccSnapshot[applied={T|T < 7333507850054914048 or (T in {7333507850054914048})}]
I20260926 07:19:49.979650 3243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:49.979831 3243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:49.985199 3243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3335d929499c4a7dae82c7ae2aa00496: Sending DeleteTablet for 3 replicas of tablet 164d37922bd54da9a897e28deed941ec
I20260926 07:19:49.986128 4140 tablet_service.cc:1514] Processing DeleteTablet for tablet 164d37922bd54da9a897e28deed941ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:49 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:45128
I20260926 07:19:49.986212 4277 tablet_service.cc:1514] Processing DeleteTablet for tablet 164d37922bd54da9a897e28deed941ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:49 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:59890
I20260926 07:19:49.987119 4535 tablet_replica.cc:330] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: stopping tablet replica
I20260926 07:19:49.987370 4414 tablet_service.cc:1514] Processing DeleteTablet for tablet 164d37922bd54da9a897e28deed941ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:49 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:48580
I20260926 07:19:49.987495 4535 raft_consensus.cc:2229] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:49.987861 4535 raft_consensus.cc:2258] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:49.988076 4536 tablet_replica.cc:330] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: stopping tablet replica
I20260926 07:19:49.988430 4536 raft_consensus.cc:2229] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:49.988814 4536 raft_consensus.cc:2258] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:49.989755 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 3211
I20260926 07:19:49.990998 4535 ts_tablet_manager.cc:1905] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:49.991551 4536 ts_tablet_manager.cc:1905] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:49.994447 4536 ts_tablet_manager.cc:1918] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:49.994668 4536 log.cc:1196] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/164d37922bd54da9a897e28deed941ec
I20260926 07:19:49.994844 4535 ts_tablet_manager.cc:1918] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:49.994973 4535 log.cc:1196] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/164d37922bd54da9a897e28deed941ec
I20260926 07:19:49.995229 4536 ts_tablet_manager.cc:1939] T 164d37922bd54da9a897e28deed941ec P 89e80c04d6c24d0983e9a0d4f71ae59f: Deleting consensus metadata
I20260926 07:19:49.995349 4535 ts_tablet_manager.cc:1939] T 164d37922bd54da9a897e28deed941ec P 76925f7ba8f842d1a98627db55b76657: Deleting consensus metadata
I20260926 07:19:49.997740 4538 tablet_replica.cc:330] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: stopping tablet replica
W20260926 07:19:49.997989 4232 connection.cc:535] server connection from 127.0.0.1:59890 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 07:19:49.998040 4093 connection.cc:535] server connection from 127.0.0.1:45128 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:19:49.999722 4538 raft_consensus.cc:2229] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:50.000247 4538 raft_consensus.cc:2258] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:50.001878 4538 ts_tablet_manager.cc:1905] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:50.004994 4538 ts_tablet_manager.cc:1918] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:50.005219 4538 log.cc:1196] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/164d37922bd54da9a897e28deed941ec
I20260926 07:19:50.005683 4538 ts_tablet_manager.cc:1939] T 164d37922bd54da9a897e28deed941ec P ab96d3b6385040429868d9fc330d6b92: Deleting consensus metadata
W20260926 07:19:50.007072 4369 connection.cc:441] server connection from 127.0.0.1:48580 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48580 (request call id 1) could send its response
I20260926 07:19:50.008790 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.988261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.28.235.253:36465)
0926 07:19:49.988508 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.988513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:49.997410 (+ 8897us) server_negotiation.cc:407] Connection header received
0926 07:19:49.997559 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:49.997566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:49.997645 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:49.997743 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:49.999340 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:50.000248 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:50.001550 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:50.001922 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:50.002031 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:50.004431 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:50.004438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:50.004443 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:50.004473 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:50.006626 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:50.007232 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:50.007238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:50.007240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:50.007341 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:50.007698 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:50.007703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:50.007705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:50.007956 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:50.008072 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:50.008443 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 07:19:50.008613 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260926 07:19:50.015805 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:50.001062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.28.235.253:36465)
0926 07:19:50.001181 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:50.001186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:50.002874 (+ 1688us) server_negotiation.cc:407] Connection header received
0926 07:19:50.003113 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:50.003120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:50.003174 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:50.003233 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:50.005606 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:50.006626 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:50.008543 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:50.008899 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:50.008992 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:50.011216 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:50.011223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:50.011226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:50.011256 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:50.013490 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:50.014307 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:50.014312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:50.014316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:50.014411 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:50.014753 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:50.014757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:50.014759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:50.014984 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:50.015108 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:50.015485 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 07:19:50.015619 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:19:50.016963 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:50.001756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.28.235.252:39391)
0926 07:19:50.001872 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:50.001877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:50.004470 (+ 2593us) server_negotiation.cc:407] Connection header received
0926 07:19:50.004678 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:50.004685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:50.004773 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:50.004840 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:50.007628 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:50.008524 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:50.010281 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:50.010680 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:50.010766 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:50.012652 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:50.012656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:50.012659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:50.012688 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:50.014583 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:50.015101 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:50.015105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:50.015107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:50.015198 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:50.015545 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:50.015549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:50.015551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:50.015774 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:50.015882 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:50.016653 (+ 771us) server_negotiation.cc:299] Negotiation successful
0926 07:19:50.016792 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:19:50.021826 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:49.988368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.28.235.252:39391)
0926 07:19:49.988614 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:49.988619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:50.011757 (+ 23138us) server_negotiation.cc:407] Connection header received
0926 07:19:50.012012 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:50.012017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:50.012091 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:50.012152 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:50.013229 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:50.014182 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:50.015429 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:50.015798 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:50.015887 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:50.017801 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:50.017806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:50.017808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:50.017837 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:50.019739 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:50.020262 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:50.020266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:50.020268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:50.020391 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:50.020727 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:50.020731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:50.020733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:50.020968 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:50.021154 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:50.021500 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 07:19:50.021632 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260926 07:19:50.971710 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:50.971145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.195:55065)
0926 07:19:50.971492 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:50.971592 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":84,"threads_started":1}
I20260926 07:19:50.971733 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:50.971157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.194:33941)
0926 07:19:50.971544 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:50.971633 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":77,"threads_started":1}
W20260926 07:19:50.972019 4481 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:34501 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
W20260926 07:19:50.972029 4344 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:34501 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
I20260926 07:19:50.972711 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:50.972335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.193:39439)
0926 07:19:50.972560 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:50.972638 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
W20260926 07:19:50.972994 4207 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:34501 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
I20260926 07:19:51.485329 4549 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3335d929499c4a7dae82c7ae2aa00496)
I20260926 07:19:51.485459 4550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3335d929499c4a7dae82c7ae2aa00496)
I20260926 07:19:51.485622 4549 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } }
I20260926 07:19:51.485723 4550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } }
I20260926 07:19:51.486699 4549 leader_election.cc:288] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501), 0259221937c64b3188c7fdcf29a0efc7 (127.28.235.252:39391)
I20260926 07:19:51.487167 4550 leader_election.cc:288] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501), ec96a06c50974233bbfdd1b858a3d28c (127.28.235.253:36465)
I20260926 07:19:51.487618 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.486928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42612)
0926 07:19:51.487417 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.487507 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":62,"threads_started":1}
W20260926 07:19:51.487876 3831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
I20260926 07:19:51.488013 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.487019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42616)
0926 07:19:51.487837 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.487910 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":62,"threads_started":1}
W20260926 07:19:51.488163 4013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
I20260926 07:19:51.488938 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.488704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42626)
0926 07:19:51.488839 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.488881 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 07:19:51.489130 3831 leader_election.cc:334] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501): Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
I20260926 07:19:51.489151 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.488888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42634)
0926 07:19:51.489041 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.489086 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 07:19:51.489255 4013 leader_election.cc:334] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501): Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
I20260926 07:19:51.491151 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.486936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:39391 (local address 127.0.0.1:47882)
0926 07:19:51.487531 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.487545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:51.487633 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:51.487968 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:51.487972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:51.487990 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:51.488090 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:51.488098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:51.489160 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:51.489163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:51.490065 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:51.490075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:51.490716 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:51.490720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:51.490812 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:51.490834 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:19:51.490934 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:19:51.491021 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260926 07:19:51.491158 4555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.487295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.28.235.252:39391)
0926 07:19:51.487644 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:51.487649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:51.487663 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:19:51.487712 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:51.487715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:51.487762 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:51.487852 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:51.488226 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:51.488998 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:51.490195 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:51.490579 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:51.490716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:51.490768 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:51.490973 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 07:19:51.491058 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":60,"threads_started":1}
I20260926 07:19:51.491382 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.486973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36465 (local address 127.0.0.1:57242)
0926 07:19:51.487416 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.487434 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:51.487560 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:51.487985 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:51.487988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:51.488006 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:51.488111 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:51.488119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:51.489280 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:51.489283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:51.490221 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:51.490230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:51.490942 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:51.490946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:51.491051 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:51.491071 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:19:51.491172 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:19:51.491249 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":64,"threads_started":1}
I20260926 07:19:51.491381 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.487262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.28.235.253:36465)
0926 07:19:51.487496 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:51.487500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:51.487517 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:19:51.487706 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:51.487712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:51.487760 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:51.487837 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:51.488248 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:51.489142 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:51.490353 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:51.490796 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:51.490935 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:51.490984 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:51.491188 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 07:19:51.491262 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260926 07:19:51.491957 3852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0259221937c64b3188c7fdcf29a0efc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec96a06c50974233bbfdd1b858a3d28c" is_pre_election: true
I20260926 07:19:51.491957 4039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec96a06c50974233bbfdd1b858a3d28c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0259221937c64b3188c7fdcf29a0efc7" is_pre_election: true
I20260926 07:19:51.492172 4039 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec96a06c50974233bbfdd1b858a3d28c in term 1.
I20260926 07:19:51.492172 3852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0259221937c64b3188c7fdcf29a0efc7 in term 1.
I20260926 07:19:51.492522 4014 leader_election.cc:302] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0259221937c64b3188c7fdcf29a0efc7, ec96a06c50974233bbfdd1b858a3d28c; no voters: 3335d929499c4a7dae82c7ae2aa00496
I20260926 07:19:51.492532 3832 leader_election.cc:302] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0259221937c64b3188c7fdcf29a0efc7, ec96a06c50974233bbfdd1b858a3d28c; no voters: 3335d929499c4a7dae82c7ae2aa00496
I20260926 07:19:51.492741 4550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 07:19:51.492799 4550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3335d929499c4a7dae82c7ae2aa00496)
I20260926 07:19:51.492792 4549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 07:19:51.492846 4550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:19:51.492849 4549 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3335d929499c4a7dae82c7ae2aa00496)
I20260926 07:19:51.492898 4549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:19:51.493925 4549 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } }
I20260926 07:19:51.493932 4550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } }
I20260926 07:19:51.494247 4549 leader_election.cc:288] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [CANDIDATE]: Term 2 election: Requested vote from peers 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501), 0259221937c64b3188c7fdcf29a0efc7 (127.28.235.252:39391)
I20260926 07:19:51.494294 4550 leader_election.cc:288] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 2 election: Requested vote from peers 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501), ec96a06c50974233bbfdd1b858a3d28c (127.28.235.253:36465)
I20260926 07:19:51.494654 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.494397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42654)
0926 07:19:51.494526 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.494592 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 07:19:51.494763 4039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec96a06c50974233bbfdd1b858a3d28c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0259221937c64b3188c7fdcf29a0efc7"
I20260926 07:19:51.494762 3852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0259221937c64b3188c7fdcf29a0efc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec96a06c50974233bbfdd1b858a3d28c"
I20260926 07:19:51.494932 4039 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ec96a06c50974233bbfdd1b858a3d28c in current term 2: Already voted for candidate 0259221937c64b3188c7fdcf29a0efc7 in this term.
I20260926 07:19:51.494971 3852 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0259221937c64b3188c7fdcf29a0efc7 in current term 2: Already voted for candidate ec96a06c50974233bbfdd1b858a3d28c in this term.
I20260926 07:19:51.495340 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.495043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42660)
0926 07:19:51.495233 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.495285 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 07:19:51.494680 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.494385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42644)
0926 07:19:51.494562 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.494616 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260926 07:19:51.495687 4013 leader_election.cc:334] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501): Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
I20260926 07:19:51.495820 4013 leader_election.cc:302] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0259221937c64b3188c7fdcf29a0efc7; no voters: 3335d929499c4a7dae82c7ae2aa00496, ec96a06c50974233bbfdd1b858a3d28c
I20260926 07:19:51.495997 4550 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 07:19:51.496287 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.495913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:42670)
0926 07:19:51.496182 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.496237 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
W20260926 07:19:51.496460 3831 leader_election.cc:334] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501): Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
I20260926 07:19:51.496542 3831 leader_election.cc:302] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec96a06c50974233bbfdd1b858a3d28c; no voters: 0259221937c64b3188c7fdcf29a0efc7, 3335d929499c4a7dae82c7ae2aa00496
I20260926 07:19:51.496683 4549 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 07:19:51.973178 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.972616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.195:50413)
0926 07:19:51.972956 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.973064 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260926 07:19:51.973178 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.972616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.194:58497)
0926 07:19:51.972956 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.973064 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260926 07:19:51.974223 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:51.973805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.193:49783)
0926 07:19:51.974097 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:51.974147 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20260926 07:19:52.974634 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:52.974130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.195:38347)
0926 07:19:52.974445 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:52.974519 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":58,"threads_started":1}
I20260926 07:19:52.974697 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:52.973919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.194:36431)
0926 07:19:52.974448 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:52.974535 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":83,"threads_started":1}
I20260926 07:19:52.975754 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:52.975286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.193:32847)
0926 07:19:52.975559 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:52.975627 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260926 07:19:53.378005 4565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:53.378187 4565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } }
I20260926 07:19:53.378646 4565 leader_election.cc:288] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501), ec96a06c50974233bbfdd1b858a3d28c (127.28.235.253:36465)
I20260926 07:19:53.379218 3852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0259221937c64b3188c7fdcf29a0efc7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec96a06c50974233bbfdd1b858a3d28c" is_pre_election: true
I20260926 07:19:53.379357 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.378927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:56564)
0926 07:19:53.379212 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.379271 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":133,"threads_started":1}
I20260926 07:19:53.379407 3852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0259221937c64b3188c7fdcf29a0efc7 in term 2.
I20260926 07:19:53.379781 4014 leader_election.cc:302] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0259221937c64b3188c7fdcf29a0efc7, ec96a06c50974233bbfdd1b858a3d28c; no voters:
I20260926 07:19:53.379976 4565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 07:19:53.380031 4565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:53.380059 4565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:19:53.380362 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.380098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:56580)
0926 07:19:53.380235 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.380299 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:19:53.380549 4013 leader_election.cc:334] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501): Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
I20260926 07:19:53.381361 4565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } }
I20260926 07:19:53.381743 4565 leader_election.cc:288] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 3 election: Requested vote from peers 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501), ec96a06c50974233bbfdd1b858a3d28c (127.28.235.253:36465)
I20260926 07:19:53.382135 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.381904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:56592)
0926 07:19:53.382004 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.382055 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:19:53.382344 3852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0259221937c64b3188c7fdcf29a0efc7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec96a06c50974233bbfdd1b858a3d28c"
I20260926 07:19:53.382478 3852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:19:53.382876 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.382651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:56594)
0926 07:19:53.382765 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.382822 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 07:19:53.383003 4013 leader_election.cc:334] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501): Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
I20260926 07:19:53.383541 3852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0259221937c64b3188c7fdcf29a0efc7 in term 3.
I20260926 07:19:53.383876 4014 leader_election.cc:302] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0259221937c64b3188c7fdcf29a0efc7, ec96a06c50974233bbfdd1b858a3d28c; no voters: 3335d929499c4a7dae82c7ae2aa00496
I20260926 07:19:53.384056 4565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:19:53.384281 4565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [term 3 LEADER]: Becoming Leader. State: Replica: 0259221937c64b3188c7fdcf29a0efc7, State: Running, Role: LEADER
I20260926 07:19:53.384563 4565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } }
I20260926 07:19:53.385787 4568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0259221937c64b3188c7fdcf29a0efc7. Latest consensus state: current_term: 3 leader_uuid: "0259221937c64b3188c7fdcf29a0efc7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:53.385912 4568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:53.386292 4570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:53.388415 4570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:53.388830 4570 catalog_manager.cc:1259] Loaded cluster ID: 2d2fff5f73d349f7a0f0a40ca3672ad4
I20260926 07:19:53.388877 4570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:53.389472 4570 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:53.389882 4570 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Loaded TSK: 0
I20260926 07:19:53.390583 4570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:19:53.403296 4029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:19:53.406500 4029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:53.409962 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.409692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:56620)
0926 07:19:53.409840 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.409894 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 07:19:53.409946 3852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [term 3 FOLLOWER]: Refusing update from remote peer 0259221937c64b3188c7fdcf29a0efc7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 07:19:53.410480 4565 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 07:19:53.410694 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.410445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.0.0.1:56636)
0926 07:19:53.410560 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.410608 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 07:19:53.410841 4013 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 -> Peer 3335d929499c4a7dae82c7ae2aa00496 (127.28.235.254:34501): Couldn't send request to peer 3335d929499c4a7dae82c7ae2aa00496. Status: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:19:53.412066 4572 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0259221937c64b3188c7fdcf29a0efc7. Latest consensus state: current_term: 3 leader_uuid: "0259221937c64b3188c7fdcf29a0efc7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:53.412225 4572 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:19:53.413048 4571 mvcc.cc:205] Tried to move back new op lower bound from 7333507864201605120 to 7333507864105725952. Current Snapshot: MvccSnapshot[applied={T|T < 7333507864201605120}]
I20260926 07:19:53.413250 4568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0259221937c64b3188c7fdcf29a0efc7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:53.413372 4568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:53.413736 4572 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0259221937c64b3188c7fdcf29a0efc7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3335d929499c4a7dae82c7ae2aa00496" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34501 } } peers { permanent_uuid: "ec96a06c50974233bbfdd1b858a3d28c" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36465 } attrs { promote: false } } peers { permanent_uuid: "0259221937c64b3188c7fdcf29a0efc7" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 39391 } attrs { promote: false } } }
I20260926 07:19:53.413913 4572 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec96a06c50974233bbfdd1b858a3d28c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:19:53.423794 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.419708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.28.235.194:41571)
0926 07:19:53.420078 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:53.420082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:53.420093 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:19:53.420156 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:53.420159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:53.420206 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:53.420288 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:53.420549 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.421347 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.422657 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.423238 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.423372 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.423425 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:53.423593 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 07:19:53.423665 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":93,"threads_started":1}
I20260926 07:19:53.423795 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.419486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:41571 (local address 127.0.0.1:56328)
0926 07:19:53.419797 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.419812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:53.419895 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:53.420307 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:53.420310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:53.420326 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:53.420433 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:53.420439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.421446 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.421452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:53.422537 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:53.422547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.423357 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.423361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:53.423451 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.423477 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:19:53.423575 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:19:53.423648 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":6,"thread_start_us":88,"threads_started":1}
I20260926 07:19:53.424206 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.419266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:35801 (local address 127.0.0.1:54748)
0926 07:19:53.419433 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.419447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:53.419532 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:53.420768 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:53.420771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:53.420790 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:53.420876 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:53.420881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.421918 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.421922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:53.423023 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:53.423031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.423892 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.423896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:53.423993 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.424026 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:19:53.424074 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:19:53.424125 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 07:19:53.424670 4277 tablet_service.cc:1467] Processing CreateTablet for tablet 3db94c9c79ea412c994665e0566bd44d (DEFAULT_TABLE table=test-table [id=340226a0c3b645f896cca1b7ff21c62a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:53.425040 4277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db94c9c79ea412c994665e0566bd44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:53.425166 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.419757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:45051 (local address 127.0.0.1:40670)
0926 07:19:53.420337 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.420348 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:53.420417 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:53.421068 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:53.421070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:53.421085 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:53.421155 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:53.421160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.422662 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.422665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:53.423765 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:53.423774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.424756 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.424760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:53.424843 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.424866 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:19:53.424944 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:19:53.425053 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":82,"threads_started":1}
I20260926 07:19:53.425257 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.420027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.28.235.195:45051)
0926 07:19:53.420589 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:53.420593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:53.420605 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:19:53.420654 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:53.420657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:53.420714 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:53.420819 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:53.421264 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.421994 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.423982 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.424633 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.424750 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.424804 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:53.424966 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 07:19:53.425089 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
I20260926 07:19:53.425828 4414 tablet_service.cc:1467] Processing CreateTablet for tablet 3db94c9c79ea412c994665e0566bd44d (DEFAULT_TABLE table=test-table [id=340226a0c3b645f896cca1b7ff21c62a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:53.426118 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db94c9c79ea412c994665e0566bd44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:53.427384 4583 tablet_bootstrap.cc:492] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: Bootstrap starting.
I20260926 07:19:53.427639 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.419355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.28.235.193:35801)
0926 07:19:53.420521 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:53.420528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:53.420544 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:19:53.420597 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:53.420603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:53.420653 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:53.420737 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:53.421044 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.421802 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.423157 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.423758 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.424416 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.427389 (+ 2973us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:53.427430 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 07:19:53.427514 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":76,"threads_started":1}
I20260926 07:19:53.428133 4140 tablet_service.cc:1467] Processing CreateTablet for tablet 3db94c9c79ea412c994665e0566bd44d (DEFAULT_TABLE table=test-table [id=340226a0c3b645f896cca1b7ff21c62a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:53.428337 4583 tablet_bootstrap.cc:654] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:53.428432 4140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db94c9c79ea412c994665e0566bd44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:53.428637 4584 tablet_bootstrap.cc:492] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: Bootstrap starting.
I20260926 07:19:53.429585 4584 tablet_bootstrap.cc:654] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:53.429795 4583 tablet_bootstrap.cc:492] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: No bootstrap required, opened a new log
I20260926 07:19:53.429925 4583 ts_tablet_manager.cc:1397] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:53.430455 4583 raft_consensus.cc:348] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } } peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } }
I20260926 07:19:53.430603 4583 raft_consensus.cc:374] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:53.430646 4583 raft_consensus.cc:729] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e80c04d6c24d0983e9a0d4f71ae59f, State: Initialized, Role: FOLLOWER
I20260926 07:19:53.430811 4583 consensus_queue.cc:260] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } } peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } }
I20260926 07:19:53.430892 4584 tablet_bootstrap.cc:492] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: No bootstrap required, opened a new log
I20260926 07:19:53.430965 4584 ts_tablet_manager.cc:1397] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:19:53.431183 4583 ts_tablet_manager.cc:1428] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:53.431396 4584 raft_consensus.cc:348] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } } peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } }
I20260926 07:19:53.431506 4584 raft_consensus.cc:374] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:53.431538 4584 raft_consensus.cc:729] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab96d3b6385040429868d9fc330d6b92, State: Initialized, Role: FOLLOWER
I20260926 07:19:53.431648 4584 consensus_queue.cc:260] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } } peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } }
I20260926 07:19:53.431847 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.431618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.194:39127)
0926 07:19:53.431719 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.431778 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 07:19:53.432004 4584 ts_tablet_manager.cc:1428] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:53.432639 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.432386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.195:56439)
0926 07:19:53.432500 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.432561 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 07:19:53.433842 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.433582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.195:34707)
0926 07:19:53.433725 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.433775 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 07:19:53.434090 4480 heartbeater.cc:503] Master 127.28.235.252:39391 was elected leader, sending a full tablet report...
I20260926 07:19:53.434300 4587 tablet_bootstrap.cc:492] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657: Bootstrap starting.
I20260926 07:19:53.434495 4343 heartbeater.cc:503] Master 127.28.235.252:39391 was elected leader, sending a full tablet report...
I20260926 07:19:53.434917 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.434665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.194:48329)
0926 07:19:53.434799 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.434850 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 07:19:53.436097 4587 tablet_bootstrap.cc:654] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:53.437315 4587 tablet_bootstrap.cc:492] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657: No bootstrap required, opened a new log
I20260926 07:19:53.437414 4587 ts_tablet_manager.cc:1397] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:53.437829 4587 raft_consensus.cc:348] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } } peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } }
I20260926 07:19:53.437920 4587 raft_consensus.cc:374] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:53.437974 4587 raft_consensus.cc:729] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76925f7ba8f842d1a98627db55b76657, State: Initialized, Role: FOLLOWER
I20260926 07:19:53.438108 4587 consensus_queue.cc:260] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } } peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } }
I20260926 07:19:53.438444 4587 ts_tablet_manager.cc:1428] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:53.438951 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.438717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.193:56225)
0926 07:19:53.438824 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.438886 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 07:19:53.439705 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.439408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.193:52389)
0926 07:19:53.439570 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.439618 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 07:19:53.440017 4206 heartbeater.cc:503] Master 127.28.235.252:39391 was elected leader, sending a full tablet report...
I20260926 07:19:53.467635 4588 raft_consensus.cc:482] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:53.467769 4588 raft_consensus.cc:504] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } } peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } }
I20260926 07:19:53.468971 4588 leader_election.cc:288] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76925f7ba8f842d1a98627db55b76657 (127.28.235.193:35801), ab96d3b6385040429868d9fc330d6b92 (127.28.235.195:45051)
I20260926 07:19:53.472823 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.469226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:35801 (local address 127.28.235.194:43717)
0926 07:19:53.469569 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.469585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:53.469725 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:53.469979 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:53.469982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:53.469999 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:53.470111 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:53.470117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.470835 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.470840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:53.471703 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:53.471711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.472412 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.472415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:53.472500 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.472522 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:19:53.472619 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:19:53.472689 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":2}
I20260926 07:19:53.472823 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.469325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:43717 (local address 127.28.235.193:35801)
0926 07:19:53.469475 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:53.469479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:53.469676 (+ 197us) server_negotiation.cc:407] Connection header received
0926 07:19:53.469821 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:53.469824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:53.469875 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:53.469955 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:53.470213 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.470727 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.471829 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.472293 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.472419 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.472473 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:53.472633 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 07:19:53.472702 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:19:53.473074 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.469383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:51887 (local address 127.28.235.195:45051)
0926 07:19:53.469530 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:53.469535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:53.469732 (+ 197us) server_negotiation.cc:407] Connection header received
0926 07:19:53.469887 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:53.469890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:53.469943 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:53.470092 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:53.470323 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.470818 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.471956 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.472481 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.472612 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.472658 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:53.472816 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:19:53.472883 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:19:53.473328 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db94c9c79ea412c994665e0566bd44d" candidate_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76925f7ba8f842d1a98627db55b76657" is_pre_election: true
I20260926 07:19:53.473413 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.469294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:45051 (local address 127.28.235.194:51887)
0926 07:19:53.469663 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.469675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:53.469763 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:53.470101 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:53.470104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:53.470120 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:53.470213 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:53.470219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.470927 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.470933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:53.471842 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:53.471851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.472604 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.472607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:53.472695 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.472724 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:19:53.472800 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 07:19:53.472867 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":131,"threads_started":1}
I20260926 07:19:53.473487 4160 raft_consensus.cc:2454] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e80c04d6c24d0983e9a0d4f71ae59f in term 0.
I20260926 07:19:53.473845 4230 leader_election.cc:302] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76925f7ba8f842d1a98627db55b76657, 89e80c04d6c24d0983e9a0d4f71ae59f; no voters:
I20260926 07:19:53.473963 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db94c9c79ea412c994665e0566bd44d" candidate_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab96d3b6385040429868d9fc330d6b92" is_pre_election: true
I20260926 07:19:53.474085 4588 raft_consensus.cc:2781] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:53.474138 4588 raft_consensus.cc:482] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:53.474184 4588 raft_consensus.cc:3037] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:53.474228 4434 raft_consensus.cc:2454] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e80c04d6c24d0983e9a0d4f71ae59f in term 0.
I20260926 07:19:53.475281 4588 raft_consensus.cc:504] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } } peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } }
I20260926 07:19:53.475678 4588 leader_election.cc:288] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [CANDIDATE]: Term 1 election: Requested vote from peers 76925f7ba8f842d1a98627db55b76657 (127.28.235.193:35801), ab96d3b6385040429868d9fc330d6b92 (127.28.235.195:45051)
I20260926 07:19:53.475968 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db94c9c79ea412c994665e0566bd44d" candidate_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76925f7ba8f842d1a98627db55b76657"
I20260926 07:19:53.476104 4160 raft_consensus.cc:3037] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:53.476051 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db94c9c79ea412c994665e0566bd44d" candidate_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab96d3b6385040429868d9fc330d6b92"
I20260926 07:19:53.476197 4434 raft_consensus.cc:3037] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:53.477059 4160 raft_consensus.cc:2454] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e80c04d6c24d0983e9a0d4f71ae59f in term 1.
I20260926 07:19:53.477356 4230 leader_election.cc:302] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76925f7ba8f842d1a98627db55b76657, 89e80c04d6c24d0983e9a0d4f71ae59f; no voters:
I20260926 07:19:53.477475 4434 raft_consensus.cc:2454] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e80c04d6c24d0983e9a0d4f71ae59f in term 1.
I20260926 07:19:53.477527 4588 raft_consensus.cc:2781] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:53.477784 4588 raft_consensus.cc:686] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 1 LEADER]: Becoming Leader. State: Replica: 89e80c04d6c24d0983e9a0d4f71ae59f, State: Running, Role: LEADER
I20260926 07:19:53.478075 4588 consensus_queue.cc:237] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } } peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } }
I20260926 07:19:53.478483 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.478272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.194:49685)
0926 07:19:53.478367 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.478419 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 07:19:53.480476 4028 catalog_manager.cc:5495] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e80c04d6c24d0983e9a0d4f71ae59f (127.28.235.194). New cstate: current_term: 1 leader_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e80c04d6c24d0983e9a0d4f71ae59f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 41571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 } health_report { overall_health: UNKNOWN } } }
W20260926 07:19:53.485742 4483 tablet.cc:2369] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:19:53.502100 4346 tablet.cc:2369] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:19:53.504456 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.500731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.28.235.194:41571)
0926 07:19:53.500864 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:53.500868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:53.500881 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:19:53.501096 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:53.501099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:53.501152 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:53.501240 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:53.501489 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.501983 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.503035 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:53.503259 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:53.503370 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:53.503644 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:53.504073 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:53.504198 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:53.504262 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 07:19:53.504320 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:19:53.507404 4160 raft_consensus.cc:1261] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657 [term 1 FOLLOWER]: Refusing update from remote peer 89e80c04d6c24d0983e9a0d4f71ae59f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:53.507591 4434 raft_consensus.cc:1261] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92 [term 1 FOLLOWER]: Refusing update from remote peer 89e80c04d6c24d0983e9a0d4f71ae59f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:53.507977 4593 consensus_queue.cc:1035] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [LEADER]: Connected to new peer: Peer: permanent_uuid: "76925f7ba8f842d1a98627db55b76657" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 35801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:53.508221 4588 consensus_queue.cc:1035] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab96d3b6385040429868d9fc330d6b92" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:53.509918 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.509580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.193:53655)
0926 07:19:53.509775 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.509839 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 07:19:53.511098 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.510803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.195:60677)
0926 07:19:53.510964 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.511021 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 07:19:53.513870 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.513578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.194:43319)
0926 07:19:53.513735 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.513792 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 07:19:53.514525 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.514212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.193:33447)
0926 07:19:53.514391 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.514447 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 07:19:53.515218 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.514969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34501 (local address 127.28.235.195:44747)
0926 07:19:53.515078 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.515141 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:34501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 07:19:53.516672 4598 mvcc.cc:205] Tried to move back new op lower bound from 7333507864600866816 to 7333507864488706048. Current Snapshot: MvccSnapshot[applied={T|T < 7333507864600866816 or (T in {7333507864600866816})}]
I20260926 07:19:53.523988 4028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:53.524199 4028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:53.528213 4028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0259221937c64b3188c7fdcf29a0efc7: Sending DeleteTablet for 3 replicas of tablet 3db94c9c79ea412c994665e0566bd44d
I20260926 07:19:53.528977 4277 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db94c9c79ea412c994665e0566bd44d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:53 UTC) from {username='oryx', principal='oryx/127.28.235.252@KRBTEST.COM'} at 127.0.0.1:56328
I20260926 07:19:53.529210 4414 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db94c9c79ea412c994665e0566bd44d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:53 UTC) from {username='oryx', principal='oryx/127.28.235.252@KRBTEST.COM'} at 127.0.0.1:40670
I20260926 07:19:53.529489 4606 tablet_replica.cc:330] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: stopping tablet replica
I20260926 07:19:53.529541 4140 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db94c9c79ea412c994665e0566bd44d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:53 UTC) from {username='oryx', principal='oryx/127.28.235.252@KRBTEST.COM'} at 127.0.0.1:54748
I20260926 07:19:53.529747 4606 raft_consensus.cc:2229] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:53.529922 4608 tablet_replica.cc:330] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657: stopping tablet replica
I20260926 07:19:53.530053 4608 raft_consensus.cc:2229] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:53.530200 4608 raft_consensus.cc:2258] T 3db94c9c79ea412c994665e0566bd44d P 76925f7ba8f842d1a98627db55b76657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:53.530283 4606 raft_consensus.cc:2258] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:53.530575 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 3989
I20260926 07:19:53.531561 4606 ts_tablet_manager.cc:1905] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:53.534273 4606 ts_tablet_manager.cc:1918] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:53.534360 4606 log.cc:1196] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/3db94c9c79ea412c994665e0566bd44d
I20260926 07:19:53.534639 4606 ts_tablet_manager.cc:1939] T 3db94c9c79ea412c994665e0566bd44d P 89e80c04d6c24d0983e9a0d4f71ae59f: Deleting consensus metadata
I20260926 07:19:53.535738 4012 catalog_manager.cc:4841] TS 89e80c04d6c24d0983e9a0d4f71ae59f (127.28.235.194:41571): tablet 3db94c9c79ea412c994665e0566bd44d (table test-table [id=340226a0c3b645f896cca1b7ff21c62a]) successfully deleted
I20260926 07:19:53.537290 4607 tablet_replica.cc:330] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: stopping tablet replica
I20260926 07:19:53.537464 4607 raft_consensus.cc:2229] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:53.537626 4607 raft_consensus.cc:2258] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:53.538524 4607 ts_tablet_manager.cc:1905] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:53.539131 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 4214
I20260926 07:19:53.539964 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:53.539294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:35801 (local address 127.0.0.1:54756)
0926 07:19:53.539797 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:53.539862 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402}
W20260926 07:19:53.541328 4011 catalog_manager.cc:4584] TS 76925f7ba8f842d1a98627db55b76657 (127.28.235.193:35801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3db94c9c79ea412c994665e0566bd44d: Network error: Client connection negotiation failed: client connection to 127.28.235.193:35801: connect: Connection refused (error 111)
I20260926 07:19:53.541433 4607 ts_tablet_manager.cc:1918] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:53.541543 4607 log.cc:1196] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/3db94c9c79ea412c994665e0566bd44d
I20260926 07:19:53.541858 4607 ts_tablet_manager.cc:1939] T 3db94c9c79ea412c994665e0566bd44d P ab96d3b6385040429868d9fc330d6b92: Deleting consensus metadata
I20260926 07:19:53.542765 4014 catalog_manager.cc:4841] TS ab96d3b6385040429868d9fc330d6b92 (127.28.235.195:45051): tablet 3db94c9c79ea412c994665e0566bd44d (table test-table [id=340226a0c3b645f896cca1b7ff21c62a]) successfully deleted
I20260926 07:19:53.546262 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 4350
I20260926 07:19:53.552960 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 3707
I20260926 07:19:53.558904 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 3814
2026-09-26T07:19:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7334 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:53 dist-test-slave-7zlw krb5kdc[4611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:53 dist-test-slave-7zlw krb5kdc[4611](info): set up 2 sockets
Sep 26 07:19:53 dist-test-slave-7zlw krb5kdc[4611](info): commencing operation
krb5kdc: starting...
W20260926 07:19:54.617875 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:54 dist-test-slave-7zlw krb5kdc[4611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.254.keytab.
Entry for principal oryx/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.254.keytab.
I20260926 07:19:54.744578 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34229
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.254.keytab
--principal=oryx/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:34229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:54.858075 4624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:54.858386 4624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:54.858429 4624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:54.862520 4624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:54.862579 4624 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:19:54.862597 4624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:54.862615 4624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:54.862632 4624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:54.862648 4624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:54.868057 4624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:34229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.254.keytab
--principal=oryx/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:54.869202 4624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:54.870205 4624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:54.879066 4629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:54.879009 4632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:54.879012 4630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:54.879601 4624 server_base.cc:1034] running on GCE node
I20260926 07:19:54.880271 4624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:54.881482 4624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:54.882684 4624 hybrid_clock.cc:648] HybridClock initialized: now 1790407194882654 us; error 46 us; skew 500 ppm
Sep 26 07:19:54 dist-test-slave-7zlw krb5kdc[4611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407194, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:54.885638 4624 init.cc:377] Logged in from keytab as oryx/127.28.235.254@KRBTEST.COM (short username oryx)
I20260926 07:19:54.886585 4624 webserver.cc:460] Webserver started at http://127.28.235.254:38537/ using document root <none> and password file <none>
I20260926 07:19:54.887236 4624 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:54.887342 4624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:54.887586 4624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:54.889788 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "c5e0e1933dad4c0c824739b827efa506"
format_stamp: "Formatted at 2026-09-26 07:19:54 on dist-test-slave-7zlw"
server_key: "ff3f4faff9618d0891a2acdb4abec9f0"
server_key_iv: "ce0cab403c99df94c5c64cfd45fbc4f6"
server_key_version: "encryptionkey@0"
I20260926 07:19:54.890378 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "c5e0e1933dad4c0c824739b827efa506"
format_stamp: "Formatted at 2026-09-26 07:19:54 on dist-test-slave-7zlw"
server_key: "ff3f4faff9618d0891a2acdb4abec9f0"
server_key_iv: "ce0cab403c99df94c5c64cfd45fbc4f6"
server_key_version: "encryptionkey@0"
I20260926 07:19:54.894568 4624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:54.897496 4639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:54.898824 4624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:54.898981 4624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "c5e0e1933dad4c0c824739b827efa506"
format_stamp: "Formatted at 2026-09-26 07:19:54 on dist-test-slave-7zlw"
server_key: "ff3f4faff9618d0891a2acdb4abec9f0"
server_key_iv: "ce0cab403c99df94c5c64cfd45fbc4f6"
server_key_version: "encryptionkey@0"
I20260926 07:19:54.899111 4624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:54.921602 4624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:54.925395 4624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:54.925617 4624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:54.934122 4624 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:34229
I20260926 07:19:54.934162 4701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:34229 every 8 connection(s)
I20260926 07:19:54.935132 4624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:54.938335 4702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:54.941718 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 4624
I20260926 07:19:54.941874 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:54.942349 29615 external_mini_cluster.cc:1428] Setting key d5156585d34ba722bb8886f16094e3da
I20260926 07:19:54.945113 4702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506: Bootstrap starting.
I20260926 07:19:54.948000 4702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:54.948942 4702 log.cc:824] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:54.951351 4702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506: No bootstrap required, opened a new log
Sep 26 07:19:54 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:54.955863 4702 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e0e1933dad4c0c824739b827efa506" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34229 } }
I20260926 07:19:54.956182 4702 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:54.956271 4702 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e0e1933dad4c0c824739b827efa506, State: Initialized, Role: FOLLOWER
I20260926 07:19:54.956754 4702 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e0e1933dad4c0c824739b827efa506" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34229 } }
I20260926 07:19:54.956872 4702 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:54.956952 4702 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:54.957072 4702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:54.958165 4702 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e0e1933dad4c0c824739b827efa506" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34229 } }
I20260926 07:19:54.958472 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:54.943978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.28.235.254:34229)
0926 07:19:54.944456 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:54.944476 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:54.944523 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:19:54.945448 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:54.945474 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:54.945904 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:54.946288 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:54.947131 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:54.948661 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:54.949738 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:54.950420 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:54.950583 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:54.953235 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:54.953241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:54.953260 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:54.953290 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:54.955228 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:54.955775 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:54.955781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:54.955787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:54.955931 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:54.956284 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:54.956288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:54.956289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:54.956927 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:54.957204 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:54.957444 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 07:19:54.957681 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":129,"threads_started":1}
I20260926 07:19:54.958508 4702 leader_election.cc:302] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5e0e1933dad4c0c824739b827efa506; no voters:
I20260926 07:19:54.958851 4702 leader_election.cc:288] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:54.958949 4707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:54.959173 4707 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [term 1 LEADER]: Becoming Leader. State: Replica: c5e0e1933dad4c0c824739b827efa506, State: Running, Role: LEADER
I20260926 07:19:54.959517 4707 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e0e1933dad4c0c824739b827efa506" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34229 } }
I20260926 07:19:54.960209 4702 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:54.961463 4708 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5e0e1933dad4c0c824739b827efa506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e0e1933dad4c0c824739b827efa506" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34229 } } }
I20260926 07:19:54.961602 4708 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:54.961890 4709 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5e0e1933dad4c0c824739b827efa506. Latest consensus state: current_term: 1 leader_uuid: "c5e0e1933dad4c0c824739b827efa506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e0e1933dad4c0c824739b827efa506" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34229 } } }
I20260926 07:19:54.961998 4709 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506 [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:54.962116 4716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:54.964838 4716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:54.970922 4716 catalog_manager.cc:1347] Generated new cluster ID: e1ebc5059ef74e08b4cf3c5fa045b338
I20260926 07:19:54.970996 4716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:54.985311 4716 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:54.986995 4716 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:55.002125 4716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506: Generated new TSK 0
I20260926 07:19:55.003242 4716 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab.
Entry for principal oryx/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab.
I20260926 07:19:55.044569 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34229
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab
--principal=oryx/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:55.180347 4728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:55.180626 4728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:55.180672 4728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:55.184790 4728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:55.184857 4728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:55.184899 4728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:19:55.184975 4728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:19:55.190336 4728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab
--principal=oryx/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.235.254:34229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:55.191459 4728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:55.192555 4728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:55.202126 4733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:55.202178 4736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:55.202178 4734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:55.202251 4728 server_base.cc:1034] running on GCE node
I20260926 07:19:55.202818 4728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:55.203433 4728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:55.204615 4728 hybrid_clock.cc:648] HybridClock initialized: now 1790407195204595 us; error 35 us; skew 500 ppm
Sep 26 07:19:55 dist-test-slave-7zlw krb5kdc[4611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407195, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:55.207660 4728 init.cc:377] Logged in from keytab as oryx/127.28.235.193@KRBTEST.COM (short username oryx)
I20260926 07:19:55.208642 4728 webserver.cc:460] Webserver started at http://127.28.235.193:39461/ using document root <none> and password file <none>
I20260926 07:19:55.209383 4728 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:55.209484 4728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:55.209769 4728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:55.211982 4728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "16d72e2869cc4d8abfa14d1df495b7a7"
format_stamp: "Formatted at 2026-09-26 07:19:55 on dist-test-slave-7zlw"
server_key: "6ccd60bf8f591eae1667e6f3b6f02393"
server_key_iv: "122436fc8c1c6a415fa3d133d0219044"
server_key_version: "encryptionkey@0"
I20260926 07:19:55.212592 4728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "16d72e2869cc4d8abfa14d1df495b7a7"
format_stamp: "Formatted at 2026-09-26 07:19:55 on dist-test-slave-7zlw"
server_key: "6ccd60bf8f591eae1667e6f3b6f02393"
server_key_iv: "122436fc8c1c6a415fa3d133d0219044"
server_key_version: "encryptionkey@0"
I20260926 07:19:55.216614 4728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:19:55.219451 4743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:55.220898 4728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:55.221153 4728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "16d72e2869cc4d8abfa14d1df495b7a7"
format_stamp: "Formatted at 2026-09-26 07:19:55 on dist-test-slave-7zlw"
server_key: "6ccd60bf8f591eae1667e6f3b6f02393"
server_key_iv: "122436fc8c1c6a415fa3d133d0219044"
server_key_version: "encryptionkey@0"
I20260926 07:19:55.221289 4728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:55.245841 4728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:55.249675 4728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:55.249853 4728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:55.251386 4728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:55.251477 4728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:55.251556 4728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:55.251612 4728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:19:55 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407195, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:55.264200 4728 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:42487
I20260926 07:19:55.264238 4858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:42487 every 8 connection(s)
I20260926 07:19:55.265319 4728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:19:55.267437 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.252404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:56645 (local address 127.28.235.254:34229)
0926 07:19:55.252538 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.252542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.253772 (+ 1230us) server_negotiation.cc:407] Connection header received
0926 07:19:55.254913 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.254919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.254987 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.255077 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:55.257036 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.258049 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.259481 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.259735 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.259857 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.263410 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:55.263417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:55.263419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:55.263454 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:55.265229 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.265798 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.265802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.265804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.265891 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.266344 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.266347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.266349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.266553 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:55.266631 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:55.267201 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 07:19:55.267318 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:19:55.268433 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.253051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34229 (local address 127.28.235.193:56645)
0926 07:19:55.253600 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:55.253647 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:55.254627 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:55.255579 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:55.255590 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:55.256153 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:55.256830 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.256853 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.258192 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.258195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.259327 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.259339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.259908 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.259912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.260137 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.260832 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:55.260856 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:55.263223 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:55.265353 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.265359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.265375 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.265687 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.265997 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.266001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.266003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.266202 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.266670 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:55.266677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:55.266976 (+ 299us) client_negotiation.cc:769] Sending connection context
0926 07:19:55.267154 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 07:19:55.267434 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":155,"threads_started":1}
I20260926 07:19:55.270100 4859 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34229
I20260926 07:19:55.270268 4859 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:55.270756 4859 heartbeater.cc:511] Master 127.28.235.254:34229 requested a full tablet report, sending...
I20260926 07:19:55.272482 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 4728
I20260926 07:19:55.272475 4655 ts_manager.cc:194] Registered new tserver with Master: 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193:42487)
I20260926 07:19:55.272624 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:19:55.273003 29615 external_mini_cluster.cc:1428] Setting key 46e74a95a57334843c4dccd99cda09b9
I20260926 07:19:55.274027 4655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:56645
I20260926 07:19:55.280745 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.272362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:33049 (local address 127.28.235.254:34229)
0926 07:19:55.272506 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.272512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.272657 (+ 145us) server_negotiation.cc:407] Connection header received
0926 07:19:55.272845 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.272850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.272905 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.273100 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:55.273608 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.274202 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.275329 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.275643 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.275721 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.277213 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:55.277217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:55.277221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:55.277250 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:55.279032 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.279476 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.279482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.279486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.279573 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.279865 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.279871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.279872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.280086 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:55.280200 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:55.280477 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:19:55.280591 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:19:55.280753 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.272256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34229 (local address 127.28.235.193:33049)
0926 07:19:55.272579 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:55.272593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:55.272726 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:55.273117 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:55.273120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:55.273334 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:55.273469 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.273477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.274362 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.274367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.275192 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.275202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.275889 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.275892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.276009 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.276583 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:55.276597 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:55.277067 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:55.279164 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.279167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.279168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.279366 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.279664 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.279667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.279668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.279754 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.280189 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:55.280193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:55.280285 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 07:19:55.280403 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 07:19:55.280552 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for oryx/127.28.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab.
Entry for principal oryx/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab.
I20260926 07:19:55.304379 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34229
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab
--principal=oryx/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:55.421283 4868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:55.421573 4868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:55.421622 4868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:55.426128 4868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:55.426200 4868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:55.426244 4868 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:19:55.426326 4868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:19:55.432080 4868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab
--principal=oryx/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.235.254:34229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:55.433348 4868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:55.434461 4868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:19:55.444715 4868 server_base.cc:1034] running on GCE node
W20260926 07:19:55.444621 4876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:55.444648 4874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:55.444671 4873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:55.445281 4868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:55.445991 4868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:55.447196 4868 hybrid_clock.cc:648] HybridClock initialized: now 1790407195447186 us; error 64 us; skew 500 ppm
Sep 26 07:19:55 dist-test-slave-7zlw krb5kdc[4611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407195, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:55.450367 4868 init.cc:377] Logged in from keytab as oryx/127.28.235.194@KRBTEST.COM (short username oryx)
I20260926 07:19:55.451285 4868 webserver.cc:460] Webserver started at http://127.28.235.194:41055/ using document root <none> and password file <none>
I20260926 07:19:55.451872 4868 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:55.451970 4868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:55.452211 4868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:55.454402 4868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "efb8a72bf0eb4e3ba83634161bdac45c"
format_stamp: "Formatted at 2026-09-26 07:19:55 on dist-test-slave-7zlw"
server_key: "49b04f43ba4556bc6dff9041abeab76d"
server_key_iv: "6189dbabdd71cdbaf88c3d7743f2264e"
server_key_version: "encryptionkey@0"
I20260926 07:19:55.454994 4868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "efb8a72bf0eb4e3ba83634161bdac45c"
format_stamp: "Formatted at 2026-09-26 07:19:55 on dist-test-slave-7zlw"
server_key: "49b04f43ba4556bc6dff9041abeab76d"
server_key_iv: "6189dbabdd71cdbaf88c3d7743f2264e"
server_key_version: "encryptionkey@0"
I20260926 07:19:55.458977 4868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:19:55.461818 4883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:55.463315 4868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:19:55.463457 4868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "efb8a72bf0eb4e3ba83634161bdac45c"
format_stamp: "Formatted at 2026-09-26 07:19:55 on dist-test-slave-7zlw"
server_key: "49b04f43ba4556bc6dff9041abeab76d"
server_key_iv: "6189dbabdd71cdbaf88c3d7743f2264e"
server_key_version: "encryptionkey@0"
I20260926 07:19:55.463587 4868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:55.482272 4868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:55.486124 4868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:55.486351 4868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:55.487918 4868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:55.487998 4868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:55.488078 4868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:55.488131 4868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:19:55 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407195, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:55.500087 4868 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:43281
I20260926 07:19:55.500142 4998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:43281 every 8 connection(s)
I20260926 07:19:55.501366 4868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:19:55.503796 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.489062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:44133 (local address 127.28.235.254:34229)
0926 07:19:55.489219 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.489224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.490252 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 07:19:55.491907 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.491915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.491967 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.492048 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:55.493882 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.494475 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.496370 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.496771 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.496860 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.499647 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:55.499651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:55.499653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:55.499680 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:55.501473 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.502103 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.502106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.502108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.502226 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.502572 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.502576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.502577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.502776 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:55.502888 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:55.503500 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 07:19:55.503628 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:19:55.504673 4892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.489567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34229 (local address 127.28.235.194:44133)
0926 07:19:55.490082 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:55.490141 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:55.491631 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:55.492296 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:55.492306 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:55.492889 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:55.493713 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.493741 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.494629 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.494635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.496226 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.496241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.496925 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.496930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.497180 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.497924 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:55.497945 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:55.499461 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:55.501608 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.501614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.501627 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.501937 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.502317 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.502320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.502322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.502460 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.502887 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:55.502894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:55.503177 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 07:19:55.503404 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 07:19:55.503735 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":109,"threads_started":1}
I20260926 07:19:55.505831 4999 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34229
I20260926 07:19:55.505970 4999 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:55.506476 4999 heartbeater.cc:511] Master 127.28.235.254:34229 requested a full tablet report, sending...
I20260926 07:19:55.507529 4656 ts_manager.cc:194] Registered new tserver with Master: efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281)
I20260926 07:19:55.508251 4656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:44133
I20260926 07:19:55.511469 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 4868
I20260926 07:19:55.511608 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:19:55.511898 29615 external_mini_cluster.cc:1428] Setting key 639a6569906f7c9647d5ba6b81c09d47
I20260926 07:19:55.516223 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.508241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:49957 (local address 127.28.235.254:34229)
0926 07:19:55.508376 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.508390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.508620 (+ 230us) server_negotiation.cc:407] Connection header received
0926 07:19:55.508730 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.508733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.508779 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.508867 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:55.509517 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.510031 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.510970 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.511230 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.511346 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.512589 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:55.512593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:55.512596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:55.512622 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:55.514235 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.514805 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.514808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.514810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.514962 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.515301 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.515306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.515308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.515552 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:55.515668 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:55.515941 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:19:55.516054 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:19:55.516238 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.508179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34229 (local address 127.28.235.194:49957)
0926 07:19:55.508528 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:55.508543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:55.508635 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:55.508948 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:55.508952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:55.509283 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:55.509396 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.509403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.510167 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.510170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.510864 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.510873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.511363 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.511366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.511470 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.512050 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:55.512062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:55.512392 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:55.514349 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.514356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.514361 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.514657 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.515094 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.515099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.515101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.515203 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.515655 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:55.515658 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:55.515767 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 07:19:55.515878 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:19:55.516045 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for oryx/127.28.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab.
Entry for principal oryx/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab.
I20260926 07:19:55.544696 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34229
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab
--principal=oryx/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:55.659735 5008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:55.660051 5008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:55.660126 5008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:55.664428 5008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:55.664526 5008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:55.664593 5008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:19:55.664677 5008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:19:55.670395 5008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab
--principal=oryx/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.235.254:34229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:55.671597 5008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:55.672645 5008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:55.682847 5016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:55.682871 5014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:55.682871 5013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:55.683066 5008 server_base.cc:1034] running on GCE node
I20260926 07:19:55.683643 5008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:55.684396 5008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:55.685588 5008 hybrid_clock.cc:648] HybridClock initialized: now 1790407195685572 us; error 35 us; skew 500 ppm
Sep 26 07:19:55 dist-test-slave-7zlw krb5kdc[4611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407195, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:55.688621 5008 init.cc:377] Logged in from keytab as oryx/127.28.235.195@KRBTEST.COM (short username oryx)
I20260926 07:19:55.689572 5008 webserver.cc:460] Webserver started at http://127.28.235.195:42047/ using document root <none> and password file <none>
I20260926 07:19:55.690176 5008 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:55.690267 5008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:55.690507 5008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:55.692626 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "30afb0b59eeb4adeb410f0c4a4cec16d"
format_stamp: "Formatted at 2026-09-26 07:19:55 on dist-test-slave-7zlw"
server_key: "02b38b5b2902b2a373f31065880be08c"
server_key_iv: "11350f541c3d44258a1851c688f7e930"
server_key_version: "encryptionkey@0"
I20260926 07:19:55.693272 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "30afb0b59eeb4adeb410f0c4a4cec16d"
format_stamp: "Formatted at 2026-09-26 07:19:55 on dist-test-slave-7zlw"
server_key: "02b38b5b2902b2a373f31065880be08c"
server_key_iv: "11350f541c3d44258a1851c688f7e930"
server_key_version: "encryptionkey@0"
I20260926 07:19:55.697237 5008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:19:55.699775 5023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:55.701141 5008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:19:55.701279 5008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "30afb0b59eeb4adeb410f0c4a4cec16d"
format_stamp: "Formatted at 2026-09-26 07:19:55 on dist-test-slave-7zlw"
server_key: "02b38b5b2902b2a373f31065880be08c"
server_key_iv: "11350f541c3d44258a1851c688f7e930"
server_key_version: "encryptionkey@0"
I20260926 07:19:55.701395 5008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:55.713867 5008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:55.717268 5008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:55.717473 5008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:55.719014 5008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:19:55.719099 5008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:55.719173 5008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:19:55.719216 5008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:19:55 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407195, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:55.730616 5008 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:45473
I20260926 07:19:55.730690 5138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:45473 every 8 connection(s)
I20260926 07:19:55.731719 5008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:19:55.734962 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.720065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:39057 (local address 127.28.235.254:34229)
0926 07:19:55.720205 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.720209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.721437 (+ 1228us) server_negotiation.cc:407] Connection header received
0926 07:19:55.722721 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.722728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.722802 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.722912 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:55.724302 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.725209 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.727300 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.727597 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.727688 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.731152 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:55.731156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:55.731159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:55.731197 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:55.732990 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.733599 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.733602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.733604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.733727 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.734021 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.734023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.734024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.734206 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:55.734282 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:55.734734 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 07:19:55.734837 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:19:55.735845 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.720557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34229 (local address 127.28.235.195:39057)
0926 07:19:55.721258 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:55.721309 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:55.722447 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:55.723205 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:55.723213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:55.723561 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:55.724145 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.724159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.725383 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.725389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.727140 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.727157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.727780 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.727785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.728038 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.728901 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:55.728930 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:55.730940 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:55.733224 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.733229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.733242 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.733499 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.733804 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.733806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.733808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.733935 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.734294 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:55.734299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:55.734528 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 07:19:55.734683 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 07:19:55.734934 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":136,"threads_started":1}
I20260926 07:19:55.737182 5139 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34229
I20260926 07:19:55.737361 5139 heartbeater.cc:464] Registering TS with master...
I20260926 07:19:55.737874 5139 heartbeater.cc:511] Master 127.28.235.254:34229 requested a full tablet report, sending...
I20260926 07:19:55.738981 4655 ts_manager.cc:194] Registered new tserver with Master: 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473)
I20260926 07:19:55.739787 4655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:39057
I20260926 07:19:55.741629 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 5008
I20260926 07:19:55.741771 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:19:55.742120 29615 external_mini_cluster.cc:1428] Setting key 2899a1710328988959d93a4fa221caa6
I20260926 07:19:55.744180 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:19:55.747342 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.739596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:44395 (local address 127.28.235.254:34229)
0926 07:19:55.739722 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.739726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.739894 (+ 168us) server_negotiation.cc:407] Connection header received
0926 07:19:55.740001 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.740004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.740067 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.740149 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:55.740658 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.741421 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.742542 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.742813 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.742929 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.744055 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:55.744059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:55.744061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:55.744085 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:55.745576 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.746103 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.746105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.746107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.746185 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.746488 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.746492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.746493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.746654 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:55.746729 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:55.747127 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 07:19:55.747229 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:19:55.747438 5145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.739471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34229 (local address 127.28.235.195:44395)
0926 07:19:55.739782 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:55.739799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:55.739905 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:55.740186 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:55.740189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:55.740424 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:55.740539 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.740545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.741575 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.741579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.742426 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.742436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.742933 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.742936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.743031 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.743585 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:55.743606 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:55.743913 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:55.745700 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.745704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.745706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.745982 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.746287 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:55.746291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:55.746294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:55.746396 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:55.746747 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:55.746752 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:55.746874 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 07:19:55.746994 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 07:19:55.747163 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
Sep 26 07:19:55 dist-test-slave-7zlw krb5kdc[4611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407195, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:19:55 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:19:55 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:19:55.769637 4705 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:19:55.762926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.28.235.254:34229)
0926 07:19:55.763063 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.763066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.763082 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:19:55.763123 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.763125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.763163 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.763228 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:55.763981 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.764537 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.765457 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.765671 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.765755 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.769463 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50100: BlockingRecv error: recv got EOF from 127.0.0.1:50100 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 07:19:55 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407195, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:55.780030 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.771321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.28.235.254:34229)
0926 07:19:55.771454 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.771457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.771583 (+ 126us) server_negotiation.cc:407] Connection header received
0926 07:19:55.771682 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.771684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.771723 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.771792 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:55.772485 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.773000 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.773878 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.774103 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.774190 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.776687 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:55.776691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:55.776693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:55.776721 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:55.778325 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.778759 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.778762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.778763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.778857 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:55.779210 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:55.779213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:55.779214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:55.779386 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:55.779507 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:55.779758 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 07:19:55.779853 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:19:55.783139 4655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:19:55.786542 4655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:55.802032 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.795665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:45473 (local address 127.0.0.1:36786)
0926 07:19:55.796594 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:55.796646 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:55.796775 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:55.797621 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:55.797629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:55.797656 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:55.797861 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.797869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.799337 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.799340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.800528 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.800537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.801461 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.801467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.801587 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.801624 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:19:55.801746 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 07:19:55.801874 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":62,"thread_start_us":92,"threads_started":1}
I20260926 07:19:55.802033 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.795844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.28.235.195:45473)
0926 07:19:55.797109 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.797122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.797142 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:19:55.797218 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.797227 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.797438 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.797591 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:55.797983 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.799204 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.800662 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.801324 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.801481 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.801644 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:55.801775 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 07:19:55.801874 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1123,"thread_start_us":102,"threads_started":1}
I20260926 07:19:55.802481 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.796474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.28.235.194:43281)
0926 07:19:55.797016 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.797050 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.797111 (+ 61us) server_negotiation.cc:407] Connection header received
0926 07:19:55.797311 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.797315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.797456 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.797577 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:55.798108 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.798847 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.800673 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.801461 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.801608 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.801718 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:55.802251 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 07:19:55.802349 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":103,"threads_started":1}
I20260926 07:19:55.802773 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.796367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.28.235.193:42487)
0926 07:19:55.797230 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:55.797236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:55.797259 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:19:55.797340 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:55.797345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:55.797511 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:55.797682 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:55.798066 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.799216 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.800568 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.801461 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.801588 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.801718 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:55.802362 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 07:19:55.802448 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":74,"threads_started":1}
I20260926 07:19:55.802979 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.795827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:42487 (local address 127.0.0.1:52892)
0926 07:19:55.796678 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:55.796692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:55.796775 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:55.797743 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:55.797746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:55.797777 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:55.797952 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.797958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.799366 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.799370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.800435 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.800445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.801784 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.801788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.801952 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.802038 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 07:19:55.802108 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 07:19:55.802165 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":84,"spinlock_wait_cycles":5632,"thread_start_us":112,"threads_started":1}
I20260926 07:19:55.803231 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:55.796035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:43281 (local address 127.0.0.1:37068)
0926 07:19:55.797000 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:55.797032 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:55.797184 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:55.797746 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:55.797748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:55.797779 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:55.797953 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.797958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.798978 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.798983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.800493 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:55.800504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:55.801784 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:55.801788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:55.801948 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:55.802009 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 07:19:55.802086 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:19:55.802165 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":145,"spinlock_wait_cycles":4096,"thread_start_us":83,"threads_started":1}
I20260926 07:19:55.805830 5073 tablet_service.cc:1467] Processing CreateTablet for tablet 57f2f94695d346b89362bd8f175d02cd (DEFAULT_TABLE table=test-table [id=40d8fa580ad6442cb82f95cb1739f481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:55.806010 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 57f2f94695d346b89362bd8f175d02cd (DEFAULT_TABLE table=test-table [id=40d8fa580ad6442cb82f95cb1739f481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:55.806485 4933 tablet_service.cc:1467] Processing CreateTablet for tablet 57f2f94695d346b89362bd8f175d02cd (DEFAULT_TABLE table=test-table [id=40d8fa580ad6442cb82f95cb1739f481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:55.807045 5073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f2f94695d346b89362bd8f175d02cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:55.807092 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f2f94695d346b89362bd8f175d02cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:55.807364 4933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f2f94695d346b89362bd8f175d02cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:55.813938 5167 tablet_bootstrap.cc:492] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: Bootstrap starting.
I20260926 07:19:55.815281 5168 tablet_bootstrap.cc:492] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: Bootstrap starting.
I20260926 07:19:55.815398 5169 tablet_bootstrap.cc:492] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: Bootstrap starting.
I20260926 07:19:55.816735 5167 tablet_bootstrap.cc:654] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:55.817153 5168 tablet_bootstrap.cc:654] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:55.817413 5169 tablet_bootstrap.cc:654] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:55.817893 5167 log.cc:824] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:55.818071 5168 log.cc:824] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:55.818297 5169 log.cc:824] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:55.820102 5167 tablet_bootstrap.cc:492] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: No bootstrap required, opened a new log
I20260926 07:19:55.820102 5169 tablet_bootstrap.cc:492] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: No bootstrap required, opened a new log
I20260926 07:19:55.820322 5167 ts_tablet_manager.cc:1397] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:19:55.820324 5169 ts_tablet_manager.cc:1397] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 07:19:55.822329 5168 tablet_bootstrap.cc:492] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: No bootstrap required, opened a new log
I20260926 07:19:55.822562 5168 ts_tablet_manager.cc:1397] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260926 07:19:55.824503 5167 raft_consensus.cc:348] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } }
I20260926 07:19:55.824502 5169 raft_consensus.cc:348] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } }
I20260926 07:19:55.824819 5169 raft_consensus.cc:374] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:55.824892 5169 raft_consensus.cc:729] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb8a72bf0eb4e3ba83634161bdac45c, State: Initialized, Role: FOLLOWER
I20260926 07:19:55.824926 5167 raft_consensus.cc:374] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:55.824992 5167 raft_consensus.cc:729] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30afb0b59eeb4adeb410f0c4a4cec16d, State: Initialized, Role: FOLLOWER
I20260926 07:19:55.825476 5169 consensus_queue.cc:260] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } }
I20260926 07:19:55.825476 5167 consensus_queue.cc:260] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } }
I20260926 07:19:55.826213 5139 heartbeater.cc:503] Master 127.28.235.254:34229 was elected leader, sending a full tablet report...
I20260926 07:19:55.826610 5167 ts_tablet_manager.cc:1428] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260926 07:19:55.826610 5169 ts_tablet_manager.cc:1428] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:19:55.826954 4999 heartbeater.cc:503] Master 127.28.235.254:34229 was elected leader, sending a full tablet report...
I20260926 07:19:55.828646 5168 raft_consensus.cc:348] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } }
I20260926 07:19:55.829097 5168 raft_consensus.cc:374] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:55.829170 5168 raft_consensus.cc:729] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d72e2869cc4d8abfa14d1df495b7a7, State: Initialized, Role: FOLLOWER
I20260926 07:19:55.829733 5168 consensus_queue.cc:260] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } }
I20260926 07:19:55.830530 4859 heartbeater.cc:503] Master 127.28.235.254:34229 was elected leader, sending a full tablet report...
I20260926 07:19:55.831017 5168 ts_tablet_manager.cc:1428] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
W20260926 07:19:55.983156 5140 tablet.cc:2369] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:19:56.003226 5000 tablet.cc:2369] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:19:56.017534 4860 tablet.cc:2369] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:19:56.187844 5175 raft_consensus.cc:482] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:56.188109 5175 raft_consensus.cc:504] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } }
I20260926 07:19:56.189589 5175 leader_election.cc:288] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473), efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281)
I20260926 07:19:56.195647 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.190079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:33019 (local address 127.28.235.194:43281)
0926 07:19:56.190278 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.190282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.190474 (+ 192us) server_negotiation.cc:407] Connection header received
0926 07:19:56.190633 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.190636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.190688 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.190774 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:56.191237 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.192081 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.194291 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.194919 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.195036 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.195094 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.195429 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.195503 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 07:19:56.195647 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.189996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:50299 (local address 127.28.235.195:45473)
0926 07:19:56.190149 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.190153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.190476 (+ 323us) server_negotiation.cc:407] Connection header received
0926 07:19:56.190632 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.190636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.190688 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.190774 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:56.191236 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.192037 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.194291 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.194901 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.195068 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.195121 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.195429 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.195503 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:19:56.195749 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.189952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:43281 (local address 127.28.235.193:33019)
0926 07:19:56.190371 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:56.190386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:56.190506 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:56.190821 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:56.190825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:56.190855 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:56.191107 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.191118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.192197 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.192201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.194061 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.194072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.195081 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.195084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.195261 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.195308 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 07:19:56.195397 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:19:56.195525 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":314,"spinlock_wait_cycles":26624,"thread_start_us":135,"threads_started":1}
I20260926 07:19:56.195760 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.189857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:45473 (local address 127.28.235.193:50299)
0926 07:19:56.190332 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:56.190349 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:56.190514 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:56.190821 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:56.190825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:56.190855 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:56.191065 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.191076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.192197 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.192201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.194003 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.194017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.195060 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.195064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.195265 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.195310 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 07:19:56.195397 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:19:56.195532 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":333,"spinlock_wait_cycles":14976,"thread_start_us":154,"threads_started":1}
I20260926 07:19:56.196593 4953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f2f94695d346b89362bd8f175d02cd" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" is_pre_election: true
I20260926 07:19:56.196604 5093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f2f94695d346b89362bd8f175d02cd" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" is_pre_election: true
I20260926 07:19:56.196929 5093 raft_consensus.cc:2454] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 0.
I20260926 07:19:56.196930 4953 raft_consensus.cc:2454] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 0.
I20260926 07:19:56.197491 4746 leader_election.cc:302] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d72e2869cc4d8abfa14d1df495b7a7, 30afb0b59eeb4adeb410f0c4a4cec16d; no voters:
I20260926 07:19:56.197760 5175 raft_consensus.cc:2781] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:56.197846 5175 raft_consensus.cc:482] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:56.197891 5175 raft_consensus.cc:3037] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.199000 5175 raft_consensus.cc:504] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } }
I20260926 07:19:56.199371 5175 leader_election.cc:288] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 election: Requested vote from peers 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473), efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281)
I20260926 07:19:56.199751 5093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f2f94695d346b89362bd8f175d02cd" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d"
I20260926 07:19:56.199762 4953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f2f94695d346b89362bd8f175d02cd" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb8a72bf0eb4e3ba83634161bdac45c"
I20260926 07:19:56.199894 5093 raft_consensus.cc:3037] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.199904 4953 raft_consensus.cc:3037] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.201326 5093 raft_consensus.cc:2454] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 1.
I20260926 07:19:56.201326 4953 raft_consensus.cc:2454] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 1.
I20260926 07:19:56.201711 4745 leader_election.cc:302] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d72e2869cc4d8abfa14d1df495b7a7, efb8a72bf0eb4e3ba83634161bdac45c; no voters:
I20260926 07:19:56.201930 5175 raft_consensus.cc:2781] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:56.202242 5175 raft_consensus.cc:686] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 LEADER]: Becoming Leader. State: Replica: 16d72e2869cc4d8abfa14d1df495b7a7, State: Running, Role: LEADER
I20260926 07:19:56.202592 5175 consensus_queue.cc:237] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } }
I20260926 07:19:56.206125 4656 catalog_manager.cc:5495] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193). New cstate: current_term: 1 leader_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:56.256428 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.252552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.28.235.193:42487)
0926 07:19:56.252681 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.252684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.252698 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:19:56.252780 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.252782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.252836 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.252933 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:56.253339 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.253896 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.254853 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.255080 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.255168 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.255490 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:56.255980 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:56.256071 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.256237 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.256301 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:19:56.263585 5093 raft_consensus.cc:1261] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Refusing update from remote peer 16d72e2869cc4d8abfa14d1df495b7a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:56.263607 4953 raft_consensus.cc:1261] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Refusing update from remote peer 16d72e2869cc4d8abfa14d1df495b7a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:56.264406 5178 consensus_queue.cc:1035] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:56.264670 5175 consensus_queue.cc:1035] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:56.273401 5182 mvcc.cc:205] Tried to move back new op lower bound from 7333507875886096384 to 7333507875648466944. Current Snapshot: MvccSnapshot[applied={T|T < 7333507875886096384}]
I20260926 07:19:56.277257 5181 mvcc.cc:205] Tried to move back new op lower bound from 7333507875886096384 to 7333507875648466944. Current Snapshot: MvccSnapshot[applied={T|T < 7333507875886096384}]
I20260926 07:19:56.285569 4655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:56.285761 4655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:56.288172 4655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506: Sending DeleteTablet for 3 replicas of tablet 57f2f94695d346b89362bd8f175d02cd
I20260926 07:19:56.288939 5073 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f2f94695d346b89362bd8f175d02cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:56 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:36786
I20260926 07:19:56.289182 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f2f94695d346b89362bd8f175d02cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:56 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:52892
I20260926 07:19:56.289465 5191 tablet_replica.cc:330] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: stopping tablet replica
I20260926 07:19:56.289660 5192 tablet_replica.cc:330] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: stopping tablet replica
I20260926 07:19:56.289626 4933 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f2f94695d346b89362bd8f175d02cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:56 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:37068
I20260926 07:19:56.289903 5191 raft_consensus.cc:2229] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:56.289945 5192 raft_consensus.cc:2229] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:56.290217 5191 raft_consensus.cc:2258] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:56.290316 5192 raft_consensus.cc:2258] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:56.291087 5193 tablet_replica.cc:330] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: stopping tablet replica
I20260926 07:19:56.291183 4655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:19:56.291479 5193 raft_consensus.cc:2229] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 07:19:56.291628 4655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:56.291795 5193 raft_consensus.cc:2258] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:56.291966 5191 ts_tablet_manager.cc:1905] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:56.293303 5193 ts_tablet_manager.cc:1905] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:56.294824 5191 ts_tablet_manager.cc:1918] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:56.295001 5191 log.cc:1196] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/57f2f94695d346b89362bd8f175d02cd
I20260926 07:19:56.295471 5191 ts_tablet_manager.cc:1939] T 57f2f94695d346b89362bd8f175d02cd P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting consensus metadata
I20260926 07:19:56.296293 5193 ts_tablet_manager.cc:1918] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:56.296470 5193 log.cc:1196] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/57f2f94695d346b89362bd8f175d02cd
I20260926 07:19:56.296823 4642 catalog_manager.cc:4841] TS 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473): tablet 57f2f94695d346b89362bd8f175d02cd (table test-table [id=40d8fa580ad6442cb82f95cb1739f481]) successfully deleted
I20260926 07:19:56.296880 5193 ts_tablet_manager.cc:1939] T 57f2f94695d346b89362bd8f175d02cd P efb8a72bf0eb4e3ba83634161bdac45c: Deleting consensus metadata
I20260926 07:19:56.298053 4933 tablet_service.cc:1467] Processing CreateTablet for tablet d8ba03dcc3964939bfe25ab8a3a98e11 (DEFAULT_TABLE table=test-table [id=945dc8f35a874c8caebaffb98195bbae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:56.298064 4793 tablet_service.cc:1467] Processing CreateTablet for tablet d8ba03dcc3964939bfe25ab8a3a98e11 (DEFAULT_TABLE table=test-table [id=945dc8f35a874c8caebaffb98195bbae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:56.298316 4933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ba03dcc3964939bfe25ab8a3a98e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.298352 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ba03dcc3964939bfe25ab8a3a98e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.299182 5192 ts_tablet_manager.cc:1905] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:56.300781 5169 tablet_bootstrap.cc:492] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c: Bootstrap starting.
I20260926 07:19:56.300806 5168 tablet_bootstrap.cc:492] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7: Bootstrap starting.
I20260926 07:19:56.301859 5169 tablet_bootstrap.cc:654] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.301882 5168 tablet_bootstrap.cc:654] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.303457 4641 catalog_manager.cc:4841] TS efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281): tablet 57f2f94695d346b89362bd8f175d02cd (table test-table [id=40d8fa580ad6442cb82f95cb1739f481]) successfully deleted
I20260926 07:19:56.303484 5192 ts_tablet_manager.cc:1918] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:56.303598 5192 log.cc:1196] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/57f2f94695d346b89362bd8f175d02cd
I20260926 07:19:56.303900 5073 tablet_service.cc:1467] Processing CreateTablet for tablet d8ba03dcc3964939bfe25ab8a3a98e11 (DEFAULT_TABLE table=test-table [id=945dc8f35a874c8caebaffb98195bbae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:56.304219 5073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ba03dcc3964939bfe25ab8a3a98e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.304242 5192 ts_tablet_manager.cc:1939] T 57f2f94695d346b89362bd8f175d02cd P 16d72e2869cc4d8abfa14d1df495b7a7: Deleting consensus metadata
I20260926 07:19:56.304316 5168 tablet_bootstrap.cc:492] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7: No bootstrap required, opened a new log
I20260926 07:19:56.304390 5168 ts_tablet_manager.cc:1397] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 07:19:56.304966 5168 raft_consensus.cc:348] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.305119 5168 raft_consensus.cc:374] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.305155 5168 raft_consensus.cc:729] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d72e2869cc4d8abfa14d1df495b7a7, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.305275 5168 consensus_queue.cc:260] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.305601 5168 ts_tablet_manager.cc:1428] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:56.305896 4643 catalog_manager.cc:4841] TS 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193:42487): tablet 57f2f94695d346b89362bd8f175d02cd (table test-table [id=40d8fa580ad6442cb82f95cb1739f481]) successfully deleted
I20260926 07:19:56.306751 5167 tablet_bootstrap.cc:492] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d: Bootstrap starting.
I20260926 07:19:56.306977 5169 tablet_bootstrap.cc:492] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c: No bootstrap required, opened a new log
I20260926 07:19:56.307076 5169 ts_tablet_manager.cc:1397] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 07:19:56.307574 5169 raft_consensus.cc:348] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.307693 5169 raft_consensus.cc:374] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.307726 5169 raft_consensus.cc:729] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb8a72bf0eb4e3ba83634161bdac45c, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.307781 5167 tablet_bootstrap.cc:654] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.307864 5169 consensus_queue.cc:260] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.308151 5169 ts_tablet_manager.cc:1428] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:19:56.310124 5167 tablet_bootstrap.cc:492] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d: No bootstrap required, opened a new log
I20260926 07:19:56.310220 5167 ts_tablet_manager.cc:1397] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260926 07:19:56.310683 5167 raft_consensus.cc:348] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.310807 5167 raft_consensus.cc:374] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.310845 5167 raft_consensus.cc:729] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30afb0b59eeb4adeb410f0c4a4cec16d, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.310971 5167 consensus_queue.cc:260] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.311702 5167 ts_tablet_manager.cc:1428] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:19:56.484845 5188 raft_consensus.cc:482] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:56.485057 5188 raft_consensus.cc:504] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.485585 5188 leader_election.cc:288] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281), 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473)
I20260926 07:19:56.486160 4953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ba03dcc3964939bfe25ab8a3a98e11" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" is_pre_election: true
I20260926 07:19:56.486171 5093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ba03dcc3964939bfe25ab8a3a98e11" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" is_pre_election: true
I20260926 07:19:56.486344 5093 raft_consensus.cc:2454] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 0.
I20260926 07:19:56.486336 4953 raft_consensus.cc:2454] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 0.
I20260926 07:19:56.486690 4745 leader_election.cc:302] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d72e2869cc4d8abfa14d1df495b7a7, efb8a72bf0eb4e3ba83634161bdac45c; no voters:
I20260926 07:19:56.486876 5188 raft_consensus.cc:2781] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:56.486999 5188 raft_consensus.cc:482] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:56.487031 5188 raft_consensus.cc:3037] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.488083 5188 raft_consensus.cc:504] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.488471 5188 leader_election.cc:288] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 election: Requested vote from peers efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281), 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473)
I20260926 07:19:56.488883 5093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ba03dcc3964939bfe25ab8a3a98e11" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d"
I20260926 07:19:56.488852 4953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ba03dcc3964939bfe25ab8a3a98e11" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb8a72bf0eb4e3ba83634161bdac45c"
I20260926 07:19:56.489019 5093 raft_consensus.cc:3037] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.489058 4953 raft_consensus.cc:3037] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.489977 5093 raft_consensus.cc:2454] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 1.
I20260926 07:19:56.490060 4953 raft_consensus.cc:2454] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 1.
I20260926 07:19:56.490329 4746 leader_election.cc:302] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d72e2869cc4d8abfa14d1df495b7a7, 30afb0b59eeb4adeb410f0c4a4cec16d; no voters:
I20260926 07:19:56.490536 5188 raft_consensus.cc:2781] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:56.490658 5188 raft_consensus.cc:686] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 LEADER]: Becoming Leader. State: Replica: 16d72e2869cc4d8abfa14d1df495b7a7, State: Running, Role: LEADER
I20260926 07:19:56.490836 5188 consensus_queue.cc:237] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.492683 4656 catalog_manager.cc:5495] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193). New cstate: current_term: 1 leader_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:19:56 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407194, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.254@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:56.527567 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.518699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34229 (local address 127.0.0.1:50118)
0926 07:19:56.519115 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:56.519132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:56.519246 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:56.519512 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:56.519516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:56.519936 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:19:56.520028 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.520035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.520789 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.520795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.521422 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.521430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.521871 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.521874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.521950 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.522552 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:19:56.522563 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:19:56.524132 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:19:56.525910 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:56.525919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:56.525926 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:56.526211 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:56.526495 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:19:56.526498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:19:56.526500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:19:56.526580 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:19:56.526957 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:19:56.526966 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:19:56.527079 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 07:19:56.527196 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 07:19:56.527350 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":127,"threads_started":1}
I20260926 07:19:56.527567 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.518807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.28.235.254:34229)
0926 07:19:56.519109 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.519117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.519191 (+ 74us) server_negotiation.cc:407] Connection header received
0926 07:19:56.519362 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.519367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.519408 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.519496 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:56.520145 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.520684 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.521539 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.521760 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.521869 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.524277 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:56.524280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:56.524282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:56.524302 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:56.525796 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:56.526309 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:56.526315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:56.526317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:56.526400 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:56.526663 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:56.526669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:56.526671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:56.526851 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:56.526957 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.527241 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.527416 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":245,"thread_start_us":86,"threads_started":1}
I20260926 07:19:56.531456 4656 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:50118:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:19:56.532135 4656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:56.537161 4933 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6cddda590f469a871ab68cfe13df30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aebd9b4a2e7b46c6a682edd3614c4a67]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:19:56.537190 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6cddda590f469a871ab68cfe13df30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aebd9b4a2e7b46c6a682edd3614c4a67]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:19:56.537338 5073 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6cddda590f469a871ab68cfe13df30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aebd9b4a2e7b46c6a682edd3614c4a67]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:19:56.537603 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6cddda590f469a871ab68cfe13df30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.537603 4933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6cddda590f469a871ab68cfe13df30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.537719 5073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6cddda590f469a871ab68cfe13df30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.540140 5167 tablet_bootstrap.cc:492] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: Bootstrap starting.
I20260926 07:19:56.540496 5168 tablet_bootstrap.cc:492] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: Bootstrap starting.
I20260926 07:19:56.541138 5169 tablet_bootstrap.cc:492] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: Bootstrap starting.
I20260926 07:19:56.541184 5167 tablet_bootstrap.cc:654] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.541391 5168 tablet_bootstrap.cc:654] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.542084 5169 tablet_bootstrap.cc:654] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.542606 5167 tablet_bootstrap.cc:492] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: No bootstrap required, opened a new log
I20260926 07:19:56.542708 5167 ts_tablet_manager.cc:1397] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:56.543130 5168 tablet_bootstrap.cc:492] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: No bootstrap required, opened a new log
I20260926 07:19:56.543224 5168 ts_tablet_manager.cc:1397] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:19:56.543239 5167 raft_consensus.cc:348] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.543355 5167 raft_consensus.cc:374] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.543390 5167 raft_consensus.cc:729] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30afb0b59eeb4adeb410f0c4a4cec16d, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.543467 5169 tablet_bootstrap.cc:492] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: No bootstrap required, opened a new log
I20260926 07:19:56.543540 5169 ts_tablet_manager.cc:1397] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:19:56.543535 5167 consensus_queue.cc:260] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.543629 5168 raft_consensus.cc:348] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.543735 5168 raft_consensus.cc:374] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.543766 5168 raft_consensus.cc:729] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d72e2869cc4d8abfa14d1df495b7a7, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.543851 5167 ts_tablet_manager.cc:1428] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:56.543883 5168 consensus_queue.cc:260] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.543962 5169 raft_consensus.cc:348] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.544054 5169 raft_consensus.cc:374] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.544085 5169 raft_consensus.cc:729] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb8a72bf0eb4e3ba83634161bdac45c, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.544085 5173 tablet_replica.cc:439] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } }
I20260926 07:19:56.544175 5168 ts_tablet_manager.cc:1428] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:56.544224 5173 tablet_replica.cc:442] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:56.544199 5169 consensus_queue.cc:260] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.544345 5188 tablet_replica.cc:439] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } }
I20260926 07:19:56.544462 5188 tablet_replica.cc:442] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:56.544638 5169 ts_tablet_manager.cc:1428] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 07:19:56.544646 5174 tablet_replica.cc:439] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } }
I20260926 07:19:56.545152 5174 tablet_replica.cc:442] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:56.686440 5174 raft_consensus.cc:482] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:56.686606 5174 raft_consensus.cc:504] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.687795 5174 leader_election.cc:288] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193:42487), 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473)
I20260926 07:19:56.692103 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.688191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:42619 (local address 127.28.235.195:45473)
0926 07:19:56.688341 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.688347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.688673 (+ 326us) server_negotiation.cc:407] Connection header received
0926 07:19:56.688904 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.688910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.688966 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.689116 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:56.689417 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.689922 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.691070 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.691490 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.691584 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.691625 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.691890 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.691969 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:19:56.692108 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.688135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:45473 (local address 127.28.235.194:42619)
0926 07:19:56.688539 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:56.688574 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:56.688724 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:56.689129 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:56.689131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:56.689146 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:56.689274 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.689278 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.690053 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.690056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.690945 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.690953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.691640 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.691645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.691747 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.691769 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:19:56.691844 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:19:56.691933 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":75,"thread_start_us":114,"threads_started":1}
I20260926 07:19:56.692145 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.688165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:37911 (local address 127.28.235.193:42487)
0926 07:19:56.688352 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.688357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.688591 (+ 234us) server_negotiation.cc:407] Connection header received
0926 07:19:56.688751 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.688757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.688818 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.688925 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:19:56.689378 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.689904 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.691171 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.691599 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.691712 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.691769 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.691951 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.692037 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 07:19:56.692130 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.688119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:42487 (local address 127.28.235.194:37911)
0926 07:19:56.688491 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:56.688503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:56.688604 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:56.689005 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:56.689037 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:56.689061 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:19:56.689257 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.689264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.690016 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.690021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.691045 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.691057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.691720 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.691723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.691804 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.691820 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:19:56.691919 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:19:56.691990 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":52,"thread_start_us":103,"threads_started":1}
I20260926 07:19:56.692672 5093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6cddda590f469a871ab68cfe13df30" candidate_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" is_pre_election: true
I20260926 07:19:56.692754 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6cddda590f469a871ab68cfe13df30" candidate_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" is_pre_election: true
I20260926 07:19:56.692833 5093 raft_consensus.cc:2454] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb8a72bf0eb4e3ba83634161bdac45c in term 0.
I20260926 07:19:56.692981 4813 raft_consensus.cc:2454] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb8a72bf0eb4e3ba83634161bdac45c in term 0.
I20260926 07:19:56.693224 4886 leader_election.cc:302] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30afb0b59eeb4adeb410f0c4a4cec16d, efb8a72bf0eb4e3ba83634161bdac45c; no voters:
I20260926 07:19:56.693492 5174 raft_consensus.cc:2781] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:56.693562 5174 raft_consensus.cc:482] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:56.693590 5174 raft_consensus.cc:3037] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.694610 5174 raft_consensus.cc:504] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.694973 5174 leader_election.cc:288] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [CANDIDATE]: Term 1 election: Requested vote from peers 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193:42487), 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473)
I20260926 07:19:56.695298 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6cddda590f469a871ab68cfe13df30" candidate_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d72e2869cc4d8abfa14d1df495b7a7"
I20260926 07:19:56.695298 5093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6cddda590f469a871ab68cfe13df30" candidate_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d"
I20260926 07:19:56.695417 4813 raft_consensus.cc:3037] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.695417 5093 raft_consensus.cc:3037] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.696483 5093 raft_consensus.cc:2454] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb8a72bf0eb4e3ba83634161bdac45c in term 1.
I20260926 07:19:56.696593 4813 raft_consensus.cc:2454] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb8a72bf0eb4e3ba83634161bdac45c in term 1.
I20260926 07:19:56.696823 4886 leader_election.cc:302] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30afb0b59eeb4adeb410f0c4a4cec16d, efb8a72bf0eb4e3ba83634161bdac45c; no voters:
I20260926 07:19:56.697050 5174 raft_consensus.cc:2781] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:56.697230 5174 raft_consensus.cc:686] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 LEADER]: Becoming Leader. State: Replica: efb8a72bf0eb4e3ba83634161bdac45c, State: Running, Role: LEADER
I20260926 07:19:56.697481 5174 consensus_queue.cc:237] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.699139 5210 tablet_replica.cc:439] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: TxnStatusTablet state changed. Reason: New leader efb8a72bf0eb4e3ba83634161bdac45c. Latest consensus state: current_term: 1 leader_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } }
I20260926 07:19:56.699251 5210 tablet_replica.cc:442] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:19:56.699717 5212 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:19:56.699892 5212 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:19:56.700344 4655 catalog_manager.cc:5495] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c reported cstate change: term changed from 0 to 1, leader changed from <none> to efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194). New cstate: current_term: 1 leader_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:56.751035 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.747167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.28.235.194:43281)
0926 07:19:56.747494 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.747504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.747518 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:19:56.747595 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.747598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.747649 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.747771 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:56.748034 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.748591 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.749510 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.749765 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.749878 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.750169 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:56.750610 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:56.750735 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.750803 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.750855 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":121,"threads_started":1}
I20260926 07:19:56.751008 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.747048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:43281 (local address 127.0.0.1:37072)
0926 07:19:56.747232 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:56.747257 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:56.747392 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:56.747791 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:56.747795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:56.747802 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:19:56.747917 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.747922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.748711 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.748714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.749395 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.749403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.749880 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.749884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.749972 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.750019 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:19:56.750706 (+ 687us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:19:56.750724 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:19:56.750794 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 07:19:56.750882 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 07:19:56.753880 5093 raft_consensus.cc:1261] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Refusing update from remote peer efb8a72bf0eb4e3ba83634161bdac45c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:56.754060 4813 raft_consensus.cc:1261] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Refusing update from remote peer efb8a72bf0eb4e3ba83634161bdac45c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:56.754428 5174 consensus_queue.cc:1035] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:56.755043 5174 consensus_queue.cc:1035] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:56.755916 5173 tablet_replica.cc:439] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: TxnStatusTablet state changed. Reason: New leader efb8a72bf0eb4e3ba83634161bdac45c. Latest consensus state: current_term: 1 leader_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } }
I20260926 07:19:56.756033 5173 tablet_replica.cc:442] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:56.756981 5188 tablet_replica.cc:439] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: TxnStatusTablet state changed. Reason: New leader efb8a72bf0eb4e3ba83634161bdac45c. Latest consensus state: current_term: 1 leader_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } }
I20260926 07:19:56.757130 5188 tablet_replica.cc:442] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:56.757416 5210 tablet_replica.cc:439] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } }
I20260926 07:19:56.757513 5210 tablet_replica.cc:442] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:19:56.758492 5173 tablet_replica.cc:439] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } }
I20260926 07:19:56.758653 5173 tablet_replica.cc:442] T 8c6cddda590f469a871ab68cfe13df30 P 30afb0b59eeb4adeb410f0c4a4cec16d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:56.760609 5183 mvcc.cc:205] Tried to move back new op lower bound from 7333507877898829824 to 7333507877675143168. Current Snapshot: MvccSnapshot[applied={T|T < 7333507877898829824 or (T in {7333507877898829824})}]
I20260926 07:19:56.763815 5174 tablet_replica.cc:439] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } }
I20260926 07:19:56.763957 5174 tablet_replica.cc:442] T 8c6cddda590f469a871ab68cfe13df30 P efb8a72bf0eb4e3ba83634161bdac45c: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:19:56.765192 5188 tablet_replica.cc:439] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } } }
I20260926 07:19:56.765318 5188 tablet_replica.cc:442] T 8c6cddda590f469a871ab68cfe13df30 P 16d72e2869cc4d8abfa14d1df495b7a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:19:56.774492 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.770671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:43281 (local address 127.28.235.193:41877)
0926 07:19:56.770974 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:56.770989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:56.771094 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:56.771375 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:56.771377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:56.771385 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:19:56.771478 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.771482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.772484 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.772486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.773208 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.773218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.773768 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.773770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.773831 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.773878 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:19:56.774178 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:19:56.774183 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:19:56.774265 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:19:56.774364 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
I20260926 07:19:56.774489 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.770729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:41877 (local address 127.28.235.194:43281)
0926 07:19:56.770874 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.770879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.771068 (+ 189us) server_negotiation.cc:407] Connection header received
0926 07:19:56.771215 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.771218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.771270 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.771351 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:56.771597 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.772381 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.773315 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.773659 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.773763 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.774004 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:56.774092 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:56.774168 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.774295 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.774350 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:19:56.781576 4953 raft_consensus.cc:1261] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Refusing update from remote peer 16d72e2869cc4d8abfa14d1df495b7a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:56.781689 5093 raft_consensus.cc:1261] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Refusing update from remote peer 16d72e2869cc4d8abfa14d1df495b7a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:56.781988 5175 consensus_queue.cc:1035] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:56.782131 5188 consensus_queue.cc:1035] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:56.804023 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.799695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:42487 (local address 127.28.235.194:33855)
0926 07:19:56.800037 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:56.800054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:19:56.800163 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:19:56.800462 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:19:56.800465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:19:56.800474 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:19:56.800594 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.800604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.801775 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.801778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.802714 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:19:56.802725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.803237 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.803241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:19:56.803328 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.803381 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:19:56.803707 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:19:56.803714 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:19:56.803781 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:19:56.803863 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":117,"threads_started":1}
I20260926 07:19:56.804023 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.799766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:33855 (local address 127.28.235.193:42487)
0926 07:19:56.799901 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.799906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.800209 (+ 303us) server_negotiation.cc:407] Connection header received
0926 07:19:56.800264 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.800267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.800319 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.800432 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:56.800735 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.801653 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.802823 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.803116 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.803217 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.803547 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:56.803626 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:56.803704 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.803799 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.803850 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:19:56.830520 4655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:56.830714 4655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:56.832883 4655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506: Sending DeleteTablet for 3 replicas of tablet d8ba03dcc3964939bfe25ab8a3a98e11
I20260926 07:19:56.833628 5073 tablet_service.cc:1514] Processing DeleteTablet for tablet d8ba03dcc3964939bfe25ab8a3a98e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:56 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:36786
I20260926 07:19:56.833915 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet d8ba03dcc3964939bfe25ab8a3a98e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:56 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:52892
I20260926 07:19:56.833938 4933 tablet_service.cc:1514] Processing DeleteTablet for tablet d8ba03dcc3964939bfe25ab8a3a98e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:56 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:37068
I20260926 07:19:56.834095 5231 tablet_replica.cc:330] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d: stopping tablet replica
I20260926 07:19:56.834229 5231 raft_consensus.cc:2229] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:56.834280 5232 tablet_replica.cc:330] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c: stopping tablet replica
I20260926 07:19:56.834388 5231 raft_consensus.cc:2258] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:56.834473 5232 raft_consensus.cc:2229] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:56.834646 5232 raft_consensus.cc:2258] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:56.835191 5231 ts_tablet_manager.cc:1905] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:56.838323 5231 ts_tablet_manager.cc:1918] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:56.838452 5231 log.cc:1196] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/d8ba03dcc3964939bfe25ab8a3a98e11
I20260926 07:19:56.838574 5233 tablet_replica.cc:330] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7: stopping tablet replica
I20260926 07:19:56.838759 5231 ts_tablet_manager.cc:1939] T d8ba03dcc3964939bfe25ab8a3a98e11 P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting consensus metadata
I20260926 07:19:56.838759 5233 raft_consensus.cc:2229] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:56.839008 5233 raft_consensus.cc:2258] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:56.839049 5232 ts_tablet_manager.cc:1905] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:56.839684 5233 ts_tablet_manager.cc:1905] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:56.839761 4642 catalog_manager.cc:4841] TS 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473): tablet d8ba03dcc3964939bfe25ab8a3a98e11 (table test-table [id=945dc8f35a874c8caebaffb98195bbae]) successfully deleted
I20260926 07:19:56.842660 5232 ts_tablet_manager.cc:1918] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:56.842836 5232 log.cc:1196] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/d8ba03dcc3964939bfe25ab8a3a98e11
I20260926 07:19:56.843245 5232 ts_tablet_manager.cc:1939] T d8ba03dcc3964939bfe25ab8a3a98e11 P efb8a72bf0eb4e3ba83634161bdac45c: Deleting consensus metadata
I20260926 07:19:56.843271 5233 ts_tablet_manager.cc:1918] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:56.843427 5233 log.cc:1196] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/d8ba03dcc3964939bfe25ab8a3a98e11
I20260926 07:19:56.843802 5233 ts_tablet_manager.cc:1939] T d8ba03dcc3964939bfe25ab8a3a98e11 P 16d72e2869cc4d8abfa14d1df495b7a7: Deleting consensus metadata
I20260926 07:19:56.844216 4641 catalog_manager.cc:4841] TS efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281): tablet d8ba03dcc3964939bfe25ab8a3a98e11 (table test-table [id=945dc8f35a874c8caebaffb98195bbae]) successfully deleted
I20260926 07:19:56.844635 4643 catalog_manager.cc:4841] TS 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193:42487): tablet d8ba03dcc3964939bfe25ab8a3a98e11 (table test-table [id=945dc8f35a874c8caebaffb98195bbae]) successfully deleted
Sep 26 07:19:56 dist-test-slave-7zlw krb5kdc[4611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:19:56 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:19:56 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:19:56.864852 5204 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:19:56.857442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.28.235.254:34229)
0926 07:19:56.857611 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.857616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.857802 (+ 186us) server_negotiation.cc:407] Connection header received
0926 07:19:56.857993 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.857996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.858082 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.858155 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:56.859085 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.859632 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.860992 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.861333 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.861415 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.864722 (+ 3307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50134: BlockingRecv error: recv got EOF from 127.0.0.1:50134 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 07:19:56 dist-test-slave-7zlw krb5kdc[4611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM
I20260926 07:19:56.876055 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.866689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.28.235.254:34229)
0926 07:19:56.866884 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.866888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.866903 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:19:56.867023 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.867026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.867072 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.867155 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:56.867823 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.868622 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.869485 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.869860 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.869966 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.872510 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:56.872514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:56.872517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:56.872548 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:56.874297 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:56.874720 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:56.874725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:56.874727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:56.874820 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:56.875178 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:56.875181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:56.875183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:56.875390 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:56.875520 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.875761 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.875887 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 07:19:56.879043 4655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:19:56.879523 4655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:56.884265 4793 tablet_service.cc:1467] Processing CreateTablet for tablet 81280156663f4f2cbd762360a5b4db14 (DEFAULT_TABLE table=test-table [id=f29c19976e0e475e83ecfe69f462918d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:56.884330 4933 tablet_service.cc:1467] Processing CreateTablet for tablet 81280156663f4f2cbd762360a5b4db14 (DEFAULT_TABLE table=test-table [id=f29c19976e0e475e83ecfe69f462918d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:56.884514 5073 tablet_service.cc:1467] Processing CreateTablet for tablet 81280156663f4f2cbd762360a5b4db14 (DEFAULT_TABLE table=test-table [id=f29c19976e0e475e83ecfe69f462918d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:56.884584 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81280156663f4f2cbd762360a5b4db14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.884626 4933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81280156663f4f2cbd762360a5b4db14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.884763 5073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81280156663f4f2cbd762360a5b4db14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.886647 5168 tablet_bootstrap.cc:492] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7: Bootstrap starting.
I20260926 07:19:56.887137 5167 tablet_bootstrap.cc:492] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d: Bootstrap starting.
I20260926 07:19:56.887209 5169 tablet_bootstrap.cc:492] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c: Bootstrap starting.
I20260926 07:19:56.887495 5168 tablet_bootstrap.cc:654] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.888053 5169 tablet_bootstrap.cc:654] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.888105 5167 tablet_bootstrap.cc:654] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.888893 5168 tablet_bootstrap.cc:492] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7: No bootstrap required, opened a new log
I20260926 07:19:56.889008 5168 ts_tablet_manager.cc:1397] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:19:56.889209 5169 tablet_bootstrap.cc:492] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c: No bootstrap required, opened a new log
I20260926 07:19:56.889305 5169 ts_tablet_manager.cc:1397] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:19:56.889345 5167 tablet_bootstrap.cc:492] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d: No bootstrap required, opened a new log
I20260926 07:19:56.889430 5167 ts_tablet_manager.cc:1397] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:19:56.889470 5168 raft_consensus.cc:348] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.889593 5168 raft_consensus.cc:374] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.889643 5168 raft_consensus.cc:729] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d72e2869cc4d8abfa14d1df495b7a7, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.889766 5168 consensus_queue.cc:260] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.889745 5169 raft_consensus.cc:348] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.889856 5169 raft_consensus.cc:374] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.889889 5169 raft_consensus.cc:729] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb8a72bf0eb4e3ba83634161bdac45c, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.889881 5167 raft_consensus.cc:348] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.889977 5167 raft_consensus.cc:374] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.890025 5167 raft_consensus.cc:729] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30afb0b59eeb4adeb410f0c4a4cec16d, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.890046 5168 ts_tablet_manager.cc:1428] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:56.890017 5169 consensus_queue.cc:260] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.890151 5167 consensus_queue.cc:260] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.890313 5169 ts_tablet_manager.cc:1428] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:19:56.890767 5167 ts_tablet_manager.cc:1428] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:19:56.934854 5229 raft_consensus.cc:482] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:56.935010 5229 raft_consensus.cc:504] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.935506 5229 leader_election.cc:288] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193:42487), 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473)
I20260926 07:19:56.935949 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81280156663f4f2cbd762360a5b4db14" candidate_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" is_pre_election: true
I20260926 07:19:56.936121 4813 raft_consensus.cc:2454] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb8a72bf0eb4e3ba83634161bdac45c in term 0.
I20260926 07:19:56.936412 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81280156663f4f2cbd762360a5b4db14" candidate_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" is_pre_election: true
I20260926 07:19:56.936447 4887 leader_election.cc:302] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d72e2869cc4d8abfa14d1df495b7a7, efb8a72bf0eb4e3ba83634161bdac45c; no voters:
I20260926 07:19:56.936558 5092 raft_consensus.cc:2454] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb8a72bf0eb4e3ba83634161bdac45c in term 0.
I20260926 07:19:56.936618 5229 raft_consensus.cc:2781] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:56.936674 5229 raft_consensus.cc:482] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:56.936715 5229 raft_consensus.cc:3037] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.937942 5229 raft_consensus.cc:504] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.938314 5229 leader_election.cc:288] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [CANDIDATE]: Term 1 election: Requested vote from peers 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193:42487), 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473)
I20260926 07:19:56.938617 4813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81280156663f4f2cbd762360a5b4db14" candidate_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d72e2869cc4d8abfa14d1df495b7a7"
I20260926 07:19:56.938761 4813 raft_consensus.cc:3037] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.938769 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81280156663f4f2cbd762360a5b4db14" candidate_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d"
I20260926 07:19:56.938886 5092 raft_consensus.cc:3037] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:56.939985 4813 raft_consensus.cc:2454] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb8a72bf0eb4e3ba83634161bdac45c in term 1.
I20260926 07:19:56.939985 5092 raft_consensus.cc:2454] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb8a72bf0eb4e3ba83634161bdac45c in term 1.
I20260926 07:19:56.940281 4886 leader_election.cc:302] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30afb0b59eeb4adeb410f0c4a4cec16d, efb8a72bf0eb4e3ba83634161bdac45c; no voters:
I20260926 07:19:56.940423 5229 raft_consensus.cc:2781] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:56.940512 5229 raft_consensus.cc:686] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 LEADER]: Becoming Leader. State: Replica: efb8a72bf0eb4e3ba83634161bdac45c, State: Running, Role: LEADER
I20260926 07:19:56.940661 5229 consensus_queue.cc:237] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.942361 4655 catalog_manager.cc:5495] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c reported cstate change: term changed from 0 to 1, leader changed from <none> to efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194). New cstate: current_term: 1 leader_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:56.956635 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:56.953495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.28.235.194:43281)
0926 07:19:56.953614 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:56.953617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:56.953651 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:19:56.953781 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:56.953783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:56.953824 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:56.953888 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:56.954212 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.954722 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.955581 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:56.955788 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:56.955858 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:56.956200 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:56.956275 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:56.956339 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:56.956472 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 07:19:56.956523 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:19:56.959162 5092 raft_consensus.cc:1261] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Refusing update from remote peer efb8a72bf0eb4e3ba83634161bdac45c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:56.959167 4813 raft_consensus.cc:1261] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Refusing update from remote peer efb8a72bf0eb4e3ba83634161bdac45c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:56.959661 5229 consensus_queue.cc:1035] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:56.959832 5218 consensus_queue.cc:1035] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:56.970182 4654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:56.970407 4654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:56.972462 4654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506: Sending DeleteTablet for 3 replicas of tablet 81280156663f4f2cbd762360a5b4db14
I20260926 07:19:56.972997 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet 81280156663f4f2cbd762360a5b4db14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:56 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:52892
I20260926 07:19:56.973153 5073 tablet_service.cc:1514] Processing DeleteTablet for tablet 81280156663f4f2cbd762360a5b4db14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:56 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:36786
I20260926 07:19:56.973230 5233 tablet_replica.cc:330] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7: stopping tablet replica
I20260926 07:19:56.973328 5231 tablet_replica.cc:330] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d: stopping tablet replica
I20260926 07:19:56.973346 5233 raft_consensus.cc:2229] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:56.973431 5231 raft_consensus.cc:2229] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:56.973485 5233 raft_consensus.cc:2258] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:56.973559 5231 raft_consensus.cc:2258] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:56.974186 4933 tablet_service.cc:1514] Processing DeleteTablet for tablet 81280156663f4f2cbd762360a5b4db14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:56 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:37068
I20260926 07:19:56.974123 4654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:19:56.974299 5231 ts_tablet_manager.cc:1905] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:56.974360 5232 tablet_replica.cc:330] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c: stopping tablet replica
I20260926 07:19:56.974463 5232 raft_consensus.cc:2229] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:56.974784 5232 raft_consensus.cc:2258] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 07:19:56.974778 4654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:19:56.975401 5232 ts_tablet_manager.cc:1905] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:56.976816 5231 ts_tablet_manager.cc:1918] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:56.976902 5231 log.cc:1196] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/81280156663f4f2cbd762360a5b4db14
I20260926 07:19:56.977208 5231 ts_tablet_manager.cc:1939] T 81280156663f4f2cbd762360a5b4db14 P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting consensus metadata
I20260926 07:19:56.977847 5233 ts_tablet_manager.cc:1905] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:56.977928 5232 ts_tablet_manager.cc:1918] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:56.978022 5232 log.cc:1196] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/81280156663f4f2cbd762360a5b4db14
I20260926 07:19:56.978264 5232 ts_tablet_manager.cc:1939] T 81280156663f4f2cbd762360a5b4db14 P efb8a72bf0eb4e3ba83634161bdac45c: Deleting consensus metadata
I20260926 07:19:56.980175 5073 tablet_service.cc:1467] Processing CreateTablet for tablet d95544be5f0548968835022083ae98a5 (DEFAULT_TABLE table=test-table [id=284015d980cf4806a1226d9ad4f30e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:56.980214 5233 ts_tablet_manager.cc:1918] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:19:56.980285 5233 log.cc:1196] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/81280156663f4f2cbd762360a5b4db14
I20260926 07:19:56.980433 5073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95544be5f0548968835022083ae98a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.980540 5233 ts_tablet_manager.cc:1939] T 81280156663f4f2cbd762360a5b4db14 P 16d72e2869cc4d8abfa14d1df495b7a7: Deleting consensus metadata
I20260926 07:19:56.982295 5167 tablet_bootstrap.cc:492] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d: Bootstrap starting.
I20260926 07:19:56.982366 4642 catalog_manager.cc:4841] TS 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473): tablet 81280156663f4f2cbd762360a5b4db14 (table test-table [id=f29c19976e0e475e83ecfe69f462918d]) successfully deleted
I20260926 07:19:56.982506 4641 catalog_manager.cc:4841] TS efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281): tablet 81280156663f4f2cbd762360a5b4db14 (table test-table [id=f29c19976e0e475e83ecfe69f462918d]) successfully deleted
I20260926 07:19:56.982904 4643 catalog_manager.cc:4841] TS 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193:42487): tablet 81280156663f4f2cbd762360a5b4db14 (table test-table [id=f29c19976e0e475e83ecfe69f462918d]) successfully deleted
I20260926 07:19:56.983219 5167 tablet_bootstrap.cc:654] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.983369 4933 tablet_service.cc:1467] Processing CreateTablet for tablet d95544be5f0548968835022083ae98a5 (DEFAULT_TABLE table=test-table [id=284015d980cf4806a1226d9ad4f30e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:56.983570 4933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95544be5f0548968835022083ae98a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.983947 4793 tablet_service.cc:1467] Processing CreateTablet for tablet d95544be5f0548968835022083ae98a5 (DEFAULT_TABLE table=test-table [id=284015d980cf4806a1226d9ad4f30e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:19:56.984194 4793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95544be5f0548968835022083ae98a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:56.985591 5169 tablet_bootstrap.cc:492] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c: Bootstrap starting.
I20260926 07:19:56.985839 5167 tablet_bootstrap.cc:492] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d: No bootstrap required, opened a new log
I20260926 07:19:56.985911 5167 ts_tablet_manager.cc:1397] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 07:19:56.986367 5168 tablet_bootstrap.cc:492] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7: Bootstrap starting.
I20260926 07:19:56.986387 5169 tablet_bootstrap.cc:654] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.986371 5167 raft_consensus.cc:348] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.986500 5167 raft_consensus.cc:374] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.986533 5167 raft_consensus.cc:729] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30afb0b59eeb4adeb410f0c4a4cec16d, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.986665 5167 consensus_queue.cc:260] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.986943 5167 ts_tablet_manager.cc:1428] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:19:56.987098 5168 tablet_bootstrap.cc:654] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:56.988406 5169 tablet_bootstrap.cc:492] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c: No bootstrap required, opened a new log
I20260926 07:19:56.988502 5169 ts_tablet_manager.cc:1397] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:19:56.988909 5169 raft_consensus.cc:348] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.989050 5169 raft_consensus.cc:374] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.989096 5169 raft_consensus.cc:729] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb8a72bf0eb4e3ba83634161bdac45c, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.989234 5169 consensus_queue.cc:260] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.989526 5169 ts_tablet_manager.cc:1428] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:19:56.992849 5168 tablet_bootstrap.cc:492] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7: No bootstrap required, opened a new log
I20260926 07:19:56.992960 5168 ts_tablet_manager.cc:1397] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260926 07:19:56.993342 5168 raft_consensus.cc:348] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.993460 5168 raft_consensus.cc:374] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:56.993505 5168 raft_consensus.cc:729] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d72e2869cc4d8abfa14d1df495b7a7, State: Initialized, Role: FOLLOWER
I20260926 07:19:56.993634 5168 consensus_queue.cc:260] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:56.993916 5168 ts_tablet_manager.cc:1428] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:19:57.000880 5227 raft_consensus.cc:482] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:19:57.000988 5227 raft_consensus.cc:504] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:57.001426 5227 leader_election.cc:288] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281), 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473)
I20260926 07:19:57.001899 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95544be5f0548968835022083ae98a5" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" is_pre_election: true
I20260926 07:19:57.002060 5092 raft_consensus.cc:2454] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 0.
I20260926 07:19:57.002367 4746 leader_election.cc:302] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d72e2869cc4d8abfa14d1df495b7a7, 30afb0b59eeb4adeb410f0c4a4cec16d; no voters:
I20260926 07:19:57.002529 5227 raft_consensus.cc:2781] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:19:57.002583 5227 raft_consensus.cc:482] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:19:57.002616 5227 raft_consensus.cc:3037] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:57.003371 4953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95544be5f0548968835022083ae98a5" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" is_pre_election: true
I20260926 07:19:57.003516 4953 raft_consensus.cc:2454] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 0.
I20260926 07:19:57.003971 5227 raft_consensus.cc:504] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:57.004335 5227 leader_election.cc:288] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 election: Requested vote from peers efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281), 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473)
I20260926 07:19:57.006086 4953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95544be5f0548968835022083ae98a5" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb8a72bf0eb4e3ba83634161bdac45c"
I20260926 07:19:57.006214 4953 raft_consensus.cc:3037] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:57.007222 4953 raft_consensus.cc:2454] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 1.
I20260926 07:19:57.007546 4745 leader_election.cc:302] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d72e2869cc4d8abfa14d1df495b7a7, efb8a72bf0eb4e3ba83634161bdac45c; no voters:
I20260926 07:19:57.007731 5227 raft_consensus.cc:2781] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:57.007807 5227 raft_consensus.cc:686] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 LEADER]: Becoming Leader. State: Replica: 16d72e2869cc4d8abfa14d1df495b7a7, State: Running, Role: LEADER
I20260926 07:19:57.007812 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95544be5f0548968835022083ae98a5" candidate_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d"
I20260926 07:19:57.007997 5092 raft_consensus.cc:3037] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:57.007966 5227 consensus_queue.cc:237] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } }
I20260926 07:19:57.009146 5092 raft_consensus.cc:2454] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d72e2869cc4d8abfa14d1df495b7a7 in term 1.
I20260926 07:19:57.010519 4653 catalog_manager.cc:5495] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193). New cstate: current_term: 1 leader_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d72e2869cc4d8abfa14d1df495b7a7" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 42487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 } health_report { overall_health: UNKNOWN } } }
I20260926 07:19:57.027935 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:57.024325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.28.235.193:42487)
0926 07:19:57.024708 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:57.024711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:57.024721 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:19:57.024757 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:57.024759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:57.024801 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:57.024885 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:19:57.025183 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:57.025817 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:57.026877 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:57.027129 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:57.027237 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:57.027486 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:19:57.027572 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:19:57.027651 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:57.027741 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 07:19:57.027795 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260926 07:19:57.034070 4953 raft_consensus.cc:1261] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Refusing update from remote peer 16d72e2869cc4d8abfa14d1df495b7a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:57.034523 5092 raft_consensus.cc:1261] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Refusing update from remote peer 16d72e2869cc4d8abfa14d1df495b7a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:19:57.034734 5230 consensus_queue.cc:1035] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb8a72bf0eb4e3ba83634161bdac45c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:57.035321 5230 consensus_queue.cc:1035] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30afb0b59eeb4adeb410f0c4a4cec16d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:19:57.069833 4653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:57.069983 4653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:19:57.072327 4653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5e0e1933dad4c0c824739b827efa506: Sending DeleteTablet for 3 replicas of tablet d95544be5f0548968835022083ae98a5
I20260926 07:19:57.072953 4933 tablet_service.cc:1514] Processing DeleteTablet for tablet d95544be5f0548968835022083ae98a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:57 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:37068
I20260926 07:19:57.072880 4793 tablet_service.cc:1514] Processing DeleteTablet for tablet d95544be5f0548968835022083ae98a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:57 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:52892
I20260926 07:19:57.073060 5073 tablet_service.cc:1514] Processing DeleteTablet for tablet d95544be5f0548968835022083ae98a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:19:57 UTC) from {username='oryx', principal='oryx/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:36786
I20260926 07:19:57.073204 5232 tablet_replica.cc:330] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c: stopping tablet replica
I20260926 07:19:57.073225 5231 tablet_replica.cc:330] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d: stopping tablet replica
I20260926 07:19:57.073330 5231 raft_consensus.cc:2229] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:57.073331 5232 raft_consensus.cc:2229] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:19:57.073457 5232 raft_consensus.cc:2258] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:57.073457 5231 raft_consensus.cc:2258] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:57.073632 5233 tablet_replica.cc:330] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7: stopping tablet replica
I20260926 07:19:57.073755 5233 raft_consensus.cc:2229] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:19:57.073933 5233 raft_consensus.cc:2258] T d95544be5f0548968835022083ae98a5 P 16d72e2869cc4d8abfa14d1df495b7a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:19:57.074636 5232 ts_tablet_manager.cc:1905] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:57.075914 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 4728
I20260926 07:19:57.077713 5231 ts_tablet_manager.cc:1905] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:19:57.080081 5231 ts_tablet_manager.cc:1918] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:57.080160 5231 log.cc:1196] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/d95544be5f0548968835022083ae98a5
I20260926 07:19:57.080413 5231 ts_tablet_manager.cc:1939] T d95544be5f0548968835022083ae98a5 P 30afb0b59eeb4adeb410f0c4a4cec16d: Deleting consensus metadata
I20260926 07:19:57.081247 4642 catalog_manager.cc:4841] TS 30afb0b59eeb4adeb410f0c4a4cec16d (127.28.235.195:45473): tablet d95544be5f0548968835022083ae98a5 (table test-table [id=284015d980cf4806a1226d9ad4f30e3a]) successfully deleted
I20260926 07:19:57.081570 5232 ts_tablet_manager.cc:1918] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:19:57.081697 5232 log.cc:1196] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/d95544be5f0548968835022083ae98a5
I20260926 07:19:57.086231 5232 ts_tablet_manager.cc:1939] T d95544be5f0548968835022083ae98a5 P efb8a72bf0eb4e3ba83634161bdac45c: Deleting consensus metadata
I20260926 07:19:57.087109 4641 catalog_manager.cc:4841] TS efb8a72bf0eb4e3ba83634161bdac45c (127.28.235.194:43281): tablet d95544be5f0548968835022083ae98a5 (table test-table [id=284015d980cf4806a1226d9ad4f30e3a]) successfully deleted
W20260926 07:19:57.087165 4643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:42487 (error 108)
I20260926 07:19:57.087797 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 4868
I20260926 07:19:57.090651 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:57.090151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:42487 (local address 127.0.0.1:52922)
0926 07:19:57.090478 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 07:19:57.090561 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
W20260926 07:19:57.090932 4643 catalog_manager.cc:4584] TS 16d72e2869cc4d8abfa14d1df495b7a7 (127.28.235.193:42487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d95544be5f0548968835022083ae98a5: Network error: Client connection negotiation failed: client connection to 127.28.235.193:42487: connect: Connection refused (error 111)
I20260926 07:19:57.095542 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 5008
I20260926 07:19:57.101613 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 4624
2026-09-26T07:19:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3541 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:57 dist-test-slave-7zlw krb5kdc[5263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:57 dist-test-slave-7zlw krb5kdc[5263](info): set up 2 sockets
Sep 26 07:19:57 dist-test-slave-7zlw krb5kdc[5263](info): commencing operation
krb5kdc: starting...
W20260926 07:19:58.161878 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:58 dist-test-slave-7zlw krb5kdc[5263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:58.292268 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:36365
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:36365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:58.405928 5276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:58.406272 5276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:58.406358 5276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:58.411367 5276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:58.411489 5276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:58.411530 5276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:58.411569 5276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:58.411603 5276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:58.417254 5276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:36365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:36365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:58.418412 5276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:58.419500 5276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:58.428249 5281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:58.428285 5282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:58.428337 5284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:58.428681 5276 server_base.cc:1034] running on GCE node
I20260926 07:19:58.429386 5276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:58.430590 5276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:58.431788 5276 hybrid_clock.cc:648] HybridClock initialized: now 1790407198431753 us; error 50 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T07:19:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1347 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:19:58 dist-test-slave-7zlw krb5kdc[5289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:19:58 dist-test-slave-7zlw krb5kdc[5289](info): set up 2 sockets
Sep 26 07:19:58 dist-test-slave-7zlw krb5kdc[5289](info): commencing operation
krb5kdc: starting...
W20260926 07:19:59.514721 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:19:59 dist-test-slave-7zlw krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:19:59.648942 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:39849
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:39849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:19:59.762725 5302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:19:59.763079 5302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:19:59.763170 5302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:19:59.767412 5302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:19:59.767517 5302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:19:59.767555 5302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:19:59.767612 5302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:19:59.767664 5302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:19:59.773540 5302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:39849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:19:59.774775 5302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:19:59.775892 5302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:19:59.784905 5307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:59.784930 5310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:19:59.784924 5308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:19:59.784977 5302 server_base.cc:1034] running on GCE node
I20260926 07:19:59.785842 5302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:19:59.787091 5302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:19:59.788293 5302 hybrid_clock.cc:648] HybridClock initialized: now 1790407199788261 us; error 53 us; skew 500 ppm
Sep 26 07:19:59 dist-test-slave-7zlw krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407199, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:19:59.791342 5302 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:19:59.792313 5302 webserver.cc:460] Webserver started at http://127.28.235.254:43261/ using document root <none> and password file <none>
I20260926 07:19:59.792951 5302 fs_manager.cc:359] Metadata directory not provided
I20260926 07:19:59.793073 5302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:19:59.793357 5302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:19:59.795610 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "09ab251b6c9d4e6d9c5859d91a4412cf"
format_stamp: "Formatted at 2026-09-26 07:19:59 on dist-test-slave-7zlw"
server_key: "c666ef0cc3a27f2a9bc8952694fac93f"
server_key_iv: "97d67759a61ee7a43f2892ab1a4d5869"
server_key_version: "encryptionkey@0"
I20260926 07:19:59.796232 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "09ab251b6c9d4e6d9c5859d91a4412cf"
format_stamp: "Formatted at 2026-09-26 07:19:59 on dist-test-slave-7zlw"
server_key: "c666ef0cc3a27f2a9bc8952694fac93f"
server_key_iv: "97d67759a61ee7a43f2892ab1a4d5869"
server_key_version: "encryptionkey@0"
I20260926 07:19:59.800267 5302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:19:59.803079 5317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:19:59.804437 5302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:19:59.804628 5302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "09ab251b6c9d4e6d9c5859d91a4412cf"
format_stamp: "Formatted at 2026-09-26 07:19:59 on dist-test-slave-7zlw"
server_key: "c666ef0cc3a27f2a9bc8952694fac93f"
server_key_iv: "97d67759a61ee7a43f2892ab1a4d5869"
server_key_version: "encryptionkey@0"
I20260926 07:19:59.804764 5302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:19:59.827801 5302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:19:59.831241 5302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:19:59.831470 5302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:19:59.839052 5302 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:39849
I20260926 07:19:59.839082 5369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:39849 every 8 connection(s)
I20260926 07:19:59.840152 5302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:19:59.843077 5370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:19:59.846056 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 5302
I20260926 07:19:59.846206 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:19:59.846521 29615 external_mini_cluster.cc:1428] Setting key ec4cc526e9885500b1e2bf0cbed0e315
I20260926 07:19:59.849630 5370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf: Bootstrap starting.
I20260926 07:19:59.851877 5370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf: Neither blocks nor log segments found. Creating new log.
I20260926 07:19:59.852640 5370 log.cc:824] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf: Log is configured to *not* fsync() on all Append() calls
I20260926 07:19:59.854797 5370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf: No bootstrap required, opened a new log
Sep 26 07:19:59 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:19:59.858351 5370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } }
I20260926 07:19:59.858623 5370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:19:59.858714 5370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ab251b6c9d4e6d9c5859d91a4412cf, State: Initialized, Role: FOLLOWER
I20260926 07:19:59.859225 5370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } }
I20260926 07:19:59.859383 5370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:19:59.859469 5370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:19:59.859573 5370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:19:59.860533 5370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } }
I20260926 07:19:59.860939 5370 leader_election.cc:302] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09ab251b6c9d4e6d9c5859d91a4412cf; no voters:
I20260926 07:19:59.861310 5370 leader_election.cc:288] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:19:59.861425 5375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:19:59.861640 5375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 1 LEADER]: Becoming Leader. State: Replica: 09ab251b6c9d4e6d9c5859d91a4412cf, State: Running, Role: LEADER
I20260926 07:19:59.862018 5375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } }
I20260926 07:19:59.862221 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:19:59.848047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.28.235.254:39849)
0926 07:19:59.848489 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 07:19:59.848504 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:19:59.848542 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:19:59.849196 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:19:59.849215 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:19:59.849564 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:19:59.849984 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:19:59.850817 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:59.851812 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:59.852728 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:19:59.853320 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:19:59.853506 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:19:59.856250 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:19:59.856256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:19:59.856271 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:19:59.856301 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:19:59.858811 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:59.859305 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:59.859317 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:59.859322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:59.859489 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:19:59.859809 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:19:59.859812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:19:59.859814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:19:59.860231 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:19:59.860436 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:19:59.860734 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 07:19:59.861108 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
I20260926 07:19:59.862673 5370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:19:59.863655 5376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } } }
I20260926 07:19:59.863767 5376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:59.863976 5376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09ab251b6c9d4e6d9c5859d91a4412cf. Latest consensus state: current_term: 1 leader_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } } }
I20260926 07:19:59.864051 5376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [sys.catalog]: This master's current role is: LEADER
I20260926 07:19:59.864446 5384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:19:59.867404 5384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:19:59.872990 5384 catalog_manager.cc:1347] Generated new cluster ID: 0fbcf38226c84388b795463b7f83a97b
I20260926 07:19:59.873108 5384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:19:59.894532 5384 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:19:59.895920 5384 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:19:59.904253 5384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf: Generated new TSK 0
I20260926 07:19:59.905253 5384 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:19:59.941387 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39849
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:20:00.059937 5396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:00.060251 5396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:00.060298 5396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:00.064815 5396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:00.064896 5396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:00.065030 5396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:20:00.070794 5396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:00.072011 5396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:00.073222 5396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:00.083515 5401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:00.083508 5404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:00.083520 5402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:00.083835 5396 server_base.cc:1034] running on GCE node
I20260926 07:20:00.084394 5396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:00.085251 5396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:00.086460 5396 hybrid_clock.cc:648] HybridClock initialized: now 1790407200086422 us; error 57 us; skew 500 ppm
Sep 26 07:20:00 dist-test-slave-7zlw krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:00.089676 5396 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:20:00.090711 5396 webserver.cc:460] Webserver started at http://127.28.235.193:43827/ using document root <none> and password file <none>
I20260926 07:20:00.091331 5396 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:00.091428 5396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:00.091670 5396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:00.094353 5396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "221eb85a91cd405497be4cda7efd7854"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "87d0ab969c3356a0a45920f12285593a"
server_key_iv: "77ed3e8644df1111f443e17160f8d00c"
server_key_version: "encryptionkey@0"
I20260926 07:20:00.095134 5396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "221eb85a91cd405497be4cda7efd7854"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "87d0ab969c3356a0a45920f12285593a"
server_key_iv: "77ed3e8644df1111f443e17160f8d00c"
server_key_version: "encryptionkey@0"
I20260926 07:20:00.099385 5396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:20:00.102073 5411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:00.103400 5396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:20:00.103552 5396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "221eb85a91cd405497be4cda7efd7854"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "87d0ab969c3356a0a45920f12285593a"
server_key_iv: "77ed3e8644df1111f443e17160f8d00c"
server_key_version: "encryptionkey@0"
I20260926 07:20:00.103673 5396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:00.120853 5396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:00.124097 5396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:00.124320 5396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:00.124979 5396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:00.125905 5396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:00.125979 5396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:00.126052 5396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:00.126106 5396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:00.135296 5396 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:33351
I20260926 07:20:00.135319 5524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:33351 every 8 connection(s)
I20260926 07:20:00.136257 5396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:20:00.138653 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 5396
I20260926 07:20:00.138811 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:20:00.139103 29615 external_mini_cluster.cc:1428] Setting key adfa81bcb6197c8a8e730adb08af7310
Sep 26 07:20:00 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:20:00.153795 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:00.138325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:34031 (local address 127.28.235.254:39849)
0926 07:20:00.138501 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:00.138506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:00.139498 (+ 992us) server_negotiation.cc:407] Connection header received
0926 07:20:00.141088 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:00.141091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:00.141151 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:00.141225 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:00.142941 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.143839 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.145192 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.145624 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.145774 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:00.148940 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:00.148947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:00.148952 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:20:00.148993 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:00.151121 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:00.151714 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:00.151721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:00.151728 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:00.151831 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:00.152210 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:00.152212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:00.152214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:00.152564 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:00.152694 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:00.153537 (+ 843us) server_negotiation.cc:299] Negotiation successful
0926 07:20:00.153662 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 07:20:00.154999 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:00.138787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:34031)
0926 07:20:00.139313 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:00.139369 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:00.140733 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:00.141449 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:00.141461 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:00.142019 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:00.142709 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:00.142726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.144044 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.144047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:00.145009 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:00.145024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.145819 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.145822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:00.146071 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:00.146811 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:00.146829 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:00.148713 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:00.151260 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:00.151266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:00.151282 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:00.151603 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:00.151932 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:00.151934 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:00.151935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:00.152097 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:00.152743 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:00.152752 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:00.153224 (+ 472us) client_negotiation.cc:769] Sending connection context
0926 07:20:00.153466 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 07:20:00.153883 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":149,"threads_started":1}
I20260926 07:20:00.156596 5525 heartbeater.cc:347] Connected to a master server at 127.28.235.254:39849
I20260926 07:20:00.156965 5525 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:00.157578 5525 heartbeater.cc:511] Master 127.28.235.254:39849 requested a full tablet report, sending...
I20260926 07:20:00.159531 5334 ts_manager.cc:194] Registered new tserver with Master: 221eb85a91cd405497be4cda7efd7854 (127.28.235.193:33351)
I20260926 07:20:00.161376 5334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:34031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:20:00.172168 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39849
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:20:00.287513 5530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:00.287797 5530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:00.287895 5530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:00.292197 5530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:00.292275 5530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:00.292412 5530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:20:00.297976 5530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:00.299211 5530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:00.300400 5530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:00.311056 5535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:00.311168 5536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:00.311244 5530 server_base.cc:1034] running on GCE node
W20260926 07:20:00.311439 5538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:00.311892 5530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:00.312623 5530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:00.313817 5530 hybrid_clock.cc:648] HybridClock initialized: now 1790407200313792 us; error 40 us; skew 500 ppm
Sep 26 07:20:00 dist-test-slave-7zlw krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:00.316751 5530 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:20:00.317734 5530 webserver.cc:460] Webserver started at http://127.28.235.194:42067/ using document root <none> and password file <none>
I20260926 07:20:00.318341 5530 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:00.318436 5530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:00.318701 5530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:00.320868 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "f6ff540f991e4215a15af9bc11d05594"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "af342066fbd79bbd1ffb05f3b3b99638"
server_key_iv: "a3d161219618d9e17b502e2793b84ee5"
server_key_version: "encryptionkey@0"
I20260926 07:20:00.321521 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "f6ff540f991e4215a15af9bc11d05594"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "af342066fbd79bbd1ffb05f3b3b99638"
server_key_iv: "a3d161219618d9e17b502e2793b84ee5"
server_key_version: "encryptionkey@0"
I20260926 07:20:00.325542 5530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:20:00.328230 5545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:00.329721 5530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:20:00.329829 5530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "f6ff540f991e4215a15af9bc11d05594"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "af342066fbd79bbd1ffb05f3b3b99638"
server_key_iv: "a3d161219618d9e17b502e2793b84ee5"
server_key_version: "encryptionkey@0"
I20260926 07:20:00.329924 5530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:00.352617 5530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:00.356012 5530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:00.356186 5530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:00.356774 5530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:00.357703 5530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:00.357755 5530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:00.357800 5530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:00.357822 5530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:00.368286 5530 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:40185
I20260926 07:20:00.368310 5658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:40185 every 8 connection(s)
I20260926 07:20:00.369360 5530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:20:00.369689 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 5530
I20260926 07:20:00.369828 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:20:00.370157 29615 external_mini_cluster.cc:1428] Setting key 851e0a4cd1fdb19735d12fd99993bc12
Sep 26 07:20:00 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
I20260926 07:20:00.386404 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:00.371505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:47823 (local address 127.28.235.254:39849)
0926 07:20:00.371636 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:00.371639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:00.372874 (+ 1235us) server_negotiation.cc:407] Connection header received
0926 07:20:00.374628 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:00.374634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:00.374702 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:00.374815 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:00.376159 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.376715 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.378784 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.379013 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.379112 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:00.382347 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:00.382351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:00.382354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:20:00.382381 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:00.383892 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:00.384468 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:00.384472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:00.384474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:00.384549 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:00.384966 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:00.384973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:00.384978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:00.385294 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:00.385385 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:00.386160 (+ 775us) server_negotiation.cc:299] Negotiation successful
0926 07:20:00.386278 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:20:00.387562 5661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:00.372107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:47823)
0926 07:20:00.372684 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:00.372750 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:00.374394 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:00.375008 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:00.375016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:00.375399 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:00.375924 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:00.375938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.376967 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.376973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:00.378636 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:00.378653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.379220 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.379227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:00.379488 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:00.380170 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:00.380192 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:00.382116 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:00.384037 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:00.384043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:00.384061 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:00.384335 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:00.384647 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:00.384651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:00.384653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:00.384814 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:00.385423 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:00.385434 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:00.385831 (+ 397us) client_negotiation.cc:769] Sending connection context
0926 07:20:00.386057 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 07:20:00.386462 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":132,"threads_started":1}
I20260926 07:20:00.388835 5659 heartbeater.cc:347] Connected to a master server at 127.28.235.254:39849
I20260926 07:20:00.389158 5659 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:00.389837 5659 heartbeater.cc:511] Master 127.28.235.254:39849 requested a full tablet report, sending...
I20260926 07:20:00.391122 5334 ts_manager.cc:194] Registered new tserver with Master: f6ff540f991e4215a15af9bc11d05594 (127.28.235.194:40185)
I20260926 07:20:00.391772 5334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:47823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:20:00.405248 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39849
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:20:00.523623 5664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:00.523902 5664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:00.523947 5664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:00.527981 5664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:00.528043 5664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:00.528141 5664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:20:00.533735 5664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:00.534914 5664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:00.535965 5664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:20:00.546182 5664 server_base.cc:1034] running on GCE node
W20260926 07:20:00.546109 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:00.546139 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:00.546109 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:00.546809 5664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:00.547499 5664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:00.548694 5664 hybrid_clock.cc:648] HybridClock initialized: now 1790407200548676 us; error 34 us; skew 500 ppm
Sep 26 07:20:00 dist-test-slave-7zlw krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:00.551669 5664 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:20:00.552521 5664 webserver.cc:460] Webserver started at http://127.28.235.195:39347/ using document root <none> and password file <none>
I20260926 07:20:00.553148 5664 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:00.553213 5664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:00.553385 5664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:00.555567 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "26c15cb97de9403fb7a90e45c9d3e18d"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "a658471fa138511a3d5af779b7558836"
server_key_iv: "33b0ff2657e09d62f5271bad1a01bd10"
server_key_version: "encryptionkey@0"
I20260926 07:20:00.556087 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "26c15cb97de9403fb7a90e45c9d3e18d"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "a658471fa138511a3d5af779b7558836"
server_key_iv: "33b0ff2657e09d62f5271bad1a01bd10"
server_key_version: "encryptionkey@0"
I20260926 07:20:00.559969 5664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:20:00.562795 5679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:00.564570 5664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:20:00.564743 5664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "26c15cb97de9403fb7a90e45c9d3e18d"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "a658471fa138511a3d5af779b7558836"
server_key_iv: "33b0ff2657e09d62f5271bad1a01bd10"
server_key_version: "encryptionkey@0"
I20260926 07:20:00.564910 5664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:00.581779 5664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:00.585466 5664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:00.585636 5664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:00.586222 5664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:00.587025 5664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:00.587081 5664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:00.587133 5664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:00.587149 5664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:00.596721 5664 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:41881
I20260926 07:20:00.596750 5792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:41881 every 8 connection(s)
I20260926 07:20:00.597882 5664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:20:00.602757 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 5664
I20260926 07:20:00.602927 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:20:00.603212 29615 external_mini_cluster.cc:1428] Setting key 8c726d358b127b301770dd539d7fa21c
Sep 26 07:20:00 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:20:00.614182 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:00.599759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:36775 (local address 127.28.235.254:39849)
0926 07:20:00.599945 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:00.599950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:00.600928 (+ 978us) server_negotiation.cc:407] Connection header received
0926 07:20:00.602681 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:00.602684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:00.602741 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:00.602867 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:00.604674 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.605597 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.607011 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.607282 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.607369 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:00.610469 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:00.610473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:00.610475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:20:00.610503 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:00.612131 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:00.612612 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:00.612615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:00.612617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:00.612696 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:00.613003 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:00.613006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:00.613007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:00.613332 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:00.613416 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:00.613905 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 07:20:00.614016 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:20:00.615123 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:00.600207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:36775)
0926 07:20:00.600752 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:00.600800 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:00.602261 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:00.603110 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:00.603119 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:00.603677 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:00.604455 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:00.604478 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.605772 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.605775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:00.606840 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:00.606851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:00.607448 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:00.607451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:00.607618 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:00.608410 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:00.608428 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:00.610298 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:00.612238 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:00.612243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:00.612257 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:00.612517 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:00.612782 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:00.612784 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:00.612785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:00.612922 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:00.613427 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:00.613434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:00.613685 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 07:20:00.613851 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 07:20:00.614130 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":115,"threads_started":1}
I20260926 07:20:00.616573 5793 heartbeater.cc:347] Connected to a master server at 127.28.235.254:39849
I20260926 07:20:00.616858 5793 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:00.617493 5793 heartbeater.cc:511] Master 127.28.235.254:39849 requested a full tablet report, sending...
I20260926 07:20:00.618609 5334 ts_manager.cc:194] Registered new tserver with Master: 26c15cb97de9403fb7a90e45c9d3e18d (127.28.235.195:41881)
I20260926 07:20:00.619202 5334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:36775
I20260926 07:20:00.629565 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab.
Entry for principal oryx/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab.
WARNING: no policy specified for oryx/127.28.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab.
Entry for principal oryx/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab.
WARNING: no policy specified for oryx/127.28.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab.
Entry for principal oryx/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab.
I20260926 07:20:00.714000 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 5396
I20260926 07:20:00.720745 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 5530
I20260926 07:20:00.726328 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 5664
I20260926 07:20:00.731626 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 5302
I20260926 07:20:00.738229 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:39849
--webserver_interface=127.28.235.254
--webserver_port=43261
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:39849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:20:00.852401 5802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:00.852690 5802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:00.852732 5802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:00.856865 5802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:20:00.856941 5802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:00.856969 5802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:00.856986 5802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:20:00.857002 5802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:20:00.862612 5802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:39849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=43261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:00.863758 5802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:00.864858 5802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:00.873839 5807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:00.873860 5808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:00.874018 5810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:00.874173 5802 server_base.cc:1034] running on GCE node
I20260926 07:20:00.874961 5802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:00.876273 5802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:00.877487 5802 hybrid_clock.cc:648] HybridClock initialized: now 1790407200877467 us; error 36 us; skew 500 ppm
Sep 26 07:20:00 dist-test-slave-7zlw krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407200, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:00.880716 5802 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:20:00.881740 5802 webserver.cc:460] Webserver started at http://127.28.235.254:43261/ using document root <none> and password file <none>
I20260926 07:20:00.882347 5802 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:00.882473 5802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:00.886528 5802 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:20:00.888890 5817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:00.890298 5802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:20:00.890466 5802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "09ab251b6c9d4e6d9c5859d91a4412cf"
format_stamp: "Formatted at 2026-09-26 07:19:59 on dist-test-slave-7zlw"
server_key: "c666ef0cc3a27f2a9bc8952694fac93f"
server_key_iv: "97d67759a61ee7a43f2892ab1a4d5869"
server_key_version: "encryptionkey@0"
I20260926 07:20:00.891112 5802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:00.916675 5802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:00.920253 5802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:00.920508 5802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:00.928169 5802 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:39849
I20260926 07:20:00.928217 5869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:39849 every 8 connection(s)
I20260926 07:20:00.929432 5802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:20:00.933894 5870 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:20:00.935483 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 5802
I20260926 07:20:00.935647 5870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf: Bootstrap starting.
I20260926 07:20:00.936007 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:33351
--local_ip_for_outbound_sockets=127.28.235.193
--tserver_master_addrs=127.28.235.254:39849
--webserver_port=43827
--webserver_interface=127.28.235.193
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.235.193
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:20:00.961776 5870 log.cc:824] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf: Log is configured to *not* fsync() on all Append() calls
I20260926 07:20:00.971148 5870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:20:00.971632 5870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf: Bootstrap complete.
I20260926 07:20:00.978060 5870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } }
I20260926 07:20:00.978654 5870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ab251b6c9d4e6d9c5859d91a4412cf, State: Initialized, Role: FOLLOWER
I20260926 07:20:00.979261 5870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } }
I20260926 07:20:00.979435 5870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:20:00.979517 5870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:20:00.979621 5870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:20:00.982448 5870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } }
I20260926 07:20:00.982848 5870 leader_election.cc:302] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09ab251b6c9d4e6d9c5859d91a4412cf; no voters:
I20260926 07:20:00.983232 5870 leader_election.cc:288] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:20:00.983385 5874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:20:00.983728 5874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [term 2 LEADER]: Becoming Leader. State: Replica: 09ab251b6c9d4e6d9c5859d91a4412cf, State: Running, Role: LEADER
I20260926 07:20:00.984145 5874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } }
I20260926 07:20:00.984741 5870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:20:00.985553 5876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09ab251b6c9d4e6d9c5859d91a4412cf. Latest consensus state: current_term: 2 leader_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } } }
I20260926 07:20:00.985671 5876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [sys.catalog]: This master's current role is: LEADER
I20260926 07:20:00.986346 5880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:20:00.986346 5875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ab251b6c9d4e6d9c5859d91a4412cf" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39849 } } }
I20260926 07:20:00.986479 5875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf [sys.catalog]: This master's current role is: LEADER
I20260926 07:20:00.989918 5880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:20:00.990818 5880 catalog_manager.cc:1259] Loaded cluster ID: 0fbcf38226c84388b795463b7f83a97b
I20260926 07:20:00.990898 5880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:20:00.992713 5880 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:20:00.993391 5880 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 09ab251b6c9d4e6d9c5859d91a4412cf: Loaded TSK: 0
I20260926 07:20:00.994444 5880 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:20:01.087536 5872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:01.087841 5872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:01.087925 5872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:01.092233 5872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:01.092340 5872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:01.092434 5872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:20:01.099541 5872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.193.keytab
--principal=oryx/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:33351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=43827
--tserver_master_addrs=127.28.235.254:39849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:01.100893 5872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:01.102063 5872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:20:01.113804 5872 server_base.cc:1034] running on GCE node
W20260926 07:20:01.113758 5897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:01.113744 5898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:01.113755 5900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:01.114434 5872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:01.115203 5872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:01.116442 5872 hybrid_clock.cc:648] HybridClock initialized: now 1790407201116380 us; error 49 us; skew 500 ppm
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407201, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:01.120110 5872 init.cc:377] Logged in from keytab as oryx/127.28.235.193@KRBTEST.COM (short username oryx)
I20260926 07:20:01.121325 5872 webserver.cc:460] Webserver started at http://127.28.235.193:43827/ using document root <none> and password file <none>
I20260926 07:20:01.122059 5872 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:01.122171 5872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:01.127482 5872 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 07:20:01.130342 5907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:01.132004 5872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:20:01.132134 5872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "221eb85a91cd405497be4cda7efd7854"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "87d0ab969c3356a0a45920f12285593a"
server_key_iv: "77ed3e8644df1111f443e17160f8d00c"
server_key_version: "encryptionkey@0"
I20260926 07:20:01.132812 5872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:01.149120 5872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:01.153213 5872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:01.153442 5872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:01.154310 5872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:01.155409 5872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:01.155486 5872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:01.155543 5872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:01.155579 5872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:01.167177 5872 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:33351
I20260926 07:20:01.167222 6020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:33351 every 8 connection(s)
I20260926 07:20:01.168349 5872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:20:01.175122 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 5872
I20260926 07:20:01.175642 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:40185
--local_ip_for_outbound_sockets=127.28.235.194
--tserver_master_addrs=127.28.235.254:39849
--webserver_port=42067
--webserver_interface=127.28.235.194
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.235.194
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:01.182973 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:01.170665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:39923)
0926 07:20:01.171187 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:01.171231 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:01.172712 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:01.173872 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:01.173878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:01.174266 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:01.174731 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.174744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.176704 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.176707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.177690 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.177700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.178805 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.178807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.178941 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.179577 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:01.179593 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:01.182187 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":159,"threads_started":1}
W20260926 07:20:01.183533 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:01.170614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:39923 (local address 127.28.235.254:39849)
0926 07:20:01.171110 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:01.171146 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:01.171347 (+ 201us) server_negotiation.cc:407] Connection header received
0926 07:20:01.173115 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:01.173126 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:01.173470 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:01.173778 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:01.174922 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.176532 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.177822 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.178643 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.178834 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.182515 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:39923: BlockingRecv error: recv got EOF from 127.28.235.193:39923 (error 108)
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":123,"threads_started":1}
W20260926 07:20:01.183708 6021 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:39849 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.235.254:39849: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:01.190418 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:01.184172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:43685)
0926 07:20:01.184318 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:01.184336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:01.184436 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:01.184738 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:01.184744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:01.184951 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:01.185139 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.185150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.185970 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.185977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.187111 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.187126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.187619 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.187622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.187730 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.188225 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:01.188239 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:01.190266 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 07:20:01.190513 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:01.184397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:43685 (local address 127.28.235.254:39849)
0926 07:20:01.184509 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:01.184515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:01.184532 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:20:01.184571 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:01.184573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:01.184612 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:01.184678 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:01.185271 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.185854 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.187243 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.187501 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.187584 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.190396 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:43685: BlockingRecv error: recv got EOF from 127.28.235.193:43685 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:01.197000 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:01.190899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:51069)
0926 07:20:01.191022 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:01.191036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:01.191140 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:01.192425 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:01.192429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:01.192600 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:01.192674 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.192679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.193466 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.193471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.194014 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.194022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.194488 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.194492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.194577 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.195085 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:01.195099 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:01.196856 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:20:01.197232 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:01.192116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:51069 (local address 127.28.235.254:39849)
0926 07:20:01.192230 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:01.192233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:01.192244 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:20:01.192277 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:01.192279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:01.192317 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:01.192409 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:01.192787 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.193359 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.194133 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.194376 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.194485 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.196982 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:51069: BlockingRecv error: recv got EOF from 127.28.235.193:51069 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 07:20:01.197329 6021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:20:01.297087 6025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:01.297355 6025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:01.297399 6025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:01.301611 6025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:01.301709 6025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:01.301815 6025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:20:01.307463 6025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.194.keytab
--principal=oryx/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:40185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=42067
--tserver_master_addrs=127.28.235.254:39849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:01.308796 6025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:01.309908 6025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:01.320245 6030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:01.320274 6031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:01.320313 6033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:01.320681 6025 server_base.cc:1034] running on GCE node
I20260926 07:20:01.321184 6025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:01.321846 6025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:01.323025 6025 hybrid_clock.cc:648] HybridClock initialized: now 1790407201323006 us; error 36 us; skew 500 ppm
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407201, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:01.326210 6025 init.cc:377] Logged in from keytab as oryx/127.28.235.194@KRBTEST.COM (short username oryx)
I20260926 07:20:01.327152 6025 webserver.cc:460] Webserver started at http://127.28.235.194:42067/ using document root <none> and password file <none>
I20260926 07:20:01.327761 6025 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:01.327841 6025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:01.332100 6025 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:20:01.334484 6040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:01.335776 6025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:20:01.335927 6025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "f6ff540f991e4215a15af9bc11d05594"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "af342066fbd79bbd1ffb05f3b3b99638"
server_key_iv: "a3d161219618d9e17b502e2793b84ee5"
server_key_version: "encryptionkey@0"
I20260926 07:20:01.336504 6025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:01.353825 6025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:01.356972 6025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:01.357231 6025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:01.357869 6025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:01.358726 6025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:01.358772 6025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:01.358846 6025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:01.358891 6025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:01.368543 6025 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:40185
I20260926 07:20:01.368562 6153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:40185 every 8 connection(s)
I20260926 07:20:01.369707 6025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:20:01.373291 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 6025
I20260926 07:20:01.373901 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:41881
--local_ip_for_outbound_sockets=127.28.235.195
--tserver_master_addrs=127.28.235.254:39849
--webserver_port=39347
--webserver_interface=127.28.235.195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.235.195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:20:01.382539 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:01.371630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:46497 (local address 127.28.235.254:39849)
0926 07:20:01.371774 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:01.371777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:01.372765 (+ 988us) server_negotiation.cc:407] Connection header received
0926 07:20:01.374871 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:01.374875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:01.374935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:01.375002 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:01.376372 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.376912 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.378210 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.378435 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.378550 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.382429 (+ 3879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:46497: BlockingRecv error: recv got EOF from 127.28.235.194:46497 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:20:01.383540 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:01.372081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:46497)
0926 07:20:01.372590 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:01.372637 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:01.374588 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:01.375246 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:01.375260 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:01.375712 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:01.376213 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.376225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.377078 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.377081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.378064 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.378073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.378592 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.378596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.378775 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.379403 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:01.379420 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:01.382320 (+ 2900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":112,"threads_started":1}
W20260926 07:20:01.384186 6154 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:39849 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.235.254:39849: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:01.390642 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:01.384657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:54899)
0926 07:20:01.384804 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:01.384822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:01.384906 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:01.385196 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:01.385198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:01.385371 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:01.385490 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.385497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.386484 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.386487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.387396 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.387406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.387903 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.387906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.388001 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.388535 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:01.388549 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:01.390508 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 07:20:01.390959 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:01.384788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:54899 (local address 127.28.235.254:39849)
0926 07:20:01.384922 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:01.384925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:01.384941 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:20:01.384989 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:01.384991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:01.385077 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:01.385153 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:01.385786 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.386369 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.387512 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.387793 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.387875 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.390629 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:54899: BlockingRecv error: recv got EOF from 127.28.235.194:54899 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:01.396776 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:01.391162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:52427)
0926 07:20:01.391272 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:01.391285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:01.391368 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:01.391720 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:01.391724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:01.391876 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:01.391961 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.391966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.392653 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.392656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.393596 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.393607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.394052 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.394055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.394137 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.394671 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:01.394683 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:01.396614 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:20:01.396836 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:01.391429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:52427 (local address 127.28.235.254:39849)
0926 07:20:01.391531 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:01.391534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:01.391544 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:20:01.391572 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:01.391574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:01.391607 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:01.391665 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:01.392068 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.392556 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.393717 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.393937 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.394023 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.396728 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:52427: BlockingRecv error: recv got EOF from 127.28.235.194:52427 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260926 07:20:01.397150 6154 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:20:01.492137 6157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:01.492416 6157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:01.492492 6157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:01.496971 6157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:01.497130 6157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:01.497226 6157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:20:01.502915 6157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/krb5kdc/oryx_127.28.235.195.keytab
--principal=oryx/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:41881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=39347
--tserver_master_addrs=127.28.235.254:39849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:01.504088 6157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:01.505285 6157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:01.515637 6163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:01.515789 6157 server_base.cc:1034] running on GCE node
W20260926 07:20:01.516055 6165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:01.515637 6162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:01.516870 6157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:01.517704 6157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:01.518898 6157 hybrid_clock.cc:648] HybridClock initialized: now 1790407201518870 us; error 40 us; skew 500 ppm
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407201, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:01.521950 6157 init.cc:377] Logged in from keytab as oryx/127.28.235.195@KRBTEST.COM (short username oryx)
I20260926 07:20:01.522819 6157 webserver.cc:460] Webserver started at http://127.28.235.195:39347/ using document root <none> and password file <none>
I20260926 07:20:01.523375 6157 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:01.523453 6157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:01.527524 6157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:20:01.529721 6172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:01.531056 6157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 07:20:01.531198 6157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "26c15cb97de9403fb7a90e45c9d3e18d"
format_stamp: "Formatted at 2026-09-26 07:20:00 on dist-test-slave-7zlw"
server_key: "a658471fa138511a3d5af779b7558836"
server_key_iv: "33b0ff2657e09d62f5271bad1a01bd10"
server_key_version: "encryptionkey@0"
I20260926 07:20:01.531785 6157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:01.555784 6157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:01.559521 6157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:01.559746 6157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:01.560395 6157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:01.561352 6157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:01.561426 6157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:01.561524 6157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:01.561579 6157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:01.572853 6157 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:41881
I20260926 07:20:01.572873 6285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:41881 every 8 connection(s)
I20260926 07:20:01.574196 6157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:20:01.581355 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 6157
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:20:01.587555 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:01.576473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:35915 (local address 127.28.235.254:39849)
0926 07:20:01.576651 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:01.576655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:01.577635 (+ 980us) server_negotiation.cc:407] Connection header received
0926 07:20:01.579110 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:01.579114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:01.579179 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:01.579296 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:01.580630 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.581294 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.582733 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.582990 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.583124 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.587390 (+ 4266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:35915: BlockingRecv error: recv got EOF from 127.28.235.195:35915 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:20:01.588169 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:01.576867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:35915)
0926 07:20:01.577418 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:01.577469 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:01.578831 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:01.579504 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:01.579512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:01.579917 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:01.580452 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.580465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.581445 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.581448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.582591 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.582607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.583179 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.583183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.583392 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.584150 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:01.584175 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:01.587212 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":145,"threads_started":1}
W20260926 07:20:01.588820 6286 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:39849 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.235.254:39849: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
I20260926 07:20:01.592971 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:01.582588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.28.235.254:39849)
0926 07:20:01.582893 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:01.582896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:01.582907 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:20:01.582941 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:01.582943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:01.582982 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:01.583093 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:01.583947 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.584804 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.586184 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.586524 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.586659 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.588313 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:01.588321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:01.588342 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:20:01.588377 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:01.590651 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:01.591185 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:01.591193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:01.591205 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:01.591334 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:01.591641 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:01.591645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:01.591647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:01.592036 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:01.592263 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:01.592574 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 07:20:01.592839 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":124,"threads_started":1}
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:01.595139 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:01.589337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:57403)
0926 07:20:01.589434 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:01.589450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:01.589530 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:01.590050 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:01.590053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:01.590218 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:01.590342 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.590349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.591077 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.591082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.591677 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.591685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.592185 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.592189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.592282 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.592872 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:01.592883 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:01.594980 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 07:20:01.595230 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:01.589720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:57403 (local address 127.28.235.254:39849)
0926 07:20:01.589848 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:01.589851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:01.589862 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:20:01.589892 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:01.589893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:01.589928 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:01.589988 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:01.590463 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.590956 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.591822 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.592061 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.592182 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.595111 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:57403: BlockingRecv error: recv got EOF from 127.28.235.195:57403 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:01 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:01.601310 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:01.595788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:43483)
0926 07:20:01.595921 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:01.595936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:01.596021 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:01.596293 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:01.596296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:01.596468 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:01.596552 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.596558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.597432 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.597435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.598008 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:01.598017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.598464 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.598467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:01.598530 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.599043 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:01.599056 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:01.601190 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 07:20:01.601526 6023 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:01.595837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:43483 (local address 127.28.235.254:39849)
0926 07:20:01.595975 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:01.595979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:01.595992 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:20:01.596102 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:01.596105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:01.596145 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:01.596245 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:01.596703 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.597319 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.598158 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:01.598366 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:01.598440 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:01.601365 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:43483: BlockingRecv error: recv got EOF from 127.28.235.195:43483 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 07:20:01.601696 6286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 07:20:02 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:02 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:02.204010 6291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:02.198126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:58293)
0926 07:20:02.198433 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:02.198446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:02.198543 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:02.198814 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:02.198816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:02.199000 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:02.199158 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:02.199165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.199968 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.199971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:02.200563 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:02.200571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.201176 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.201179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:02.201249 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:02.201732 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:02.201744 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:02.203812 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
W20260926 07:20:02.204092 6292 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:02.198198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:58293 (local address 127.28.235.254:39849)
0926 07:20:02.198485 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:02.198488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:02.198511 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:20:02.198643 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:02.198646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:02.198692 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:02.198773 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:02.199312 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.199867 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.200681 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.200908 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.201131 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:02.203950 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:58293: BlockingRecv error: recv got EOF from 127.28.235.193:58293 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
Sep 26 07:20:02 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:02 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:02.404083 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:02.397948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:55613)
0926 07:20:02.398260 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:02.398274 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:02.398379 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:02.398704 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:02.398707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:02.398920 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:02.399040 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:02.399046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.399946 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.399949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:02.400567 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:02.400576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.401137 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.401142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:02.401240 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:02.401882 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:02.401908 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:02.403954 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
W20260926 07:20:02.404434 6292 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:02.398066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:55613 (local address 127.28.235.254:39849)
0926 07:20:02.398239 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:02.398243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:02.398431 (+ 188us) server_negotiation.cc:407] Connection header received
0926 07:20:02.398491 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:02.398493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:02.398547 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:02.398638 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:02.399198 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.399819 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.400688 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.400909 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.401000 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:02.404324 (+ 3324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:55613: BlockingRecv error: recv got EOF from 127.28.235.194:55613 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 07:20:02 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:02 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:02.608440 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:02.602364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:51767)
0926 07:20:02.602722 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:02.602736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:02.602837 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:02.603201 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:02.603204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:02.603406 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:02.603524 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:02.603530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.604335 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.604338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:02.605183 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:02.605195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.605667 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.605670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:02.605743 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:02.606371 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:02.606383 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:02.608311 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":139,"threads_started":1}
W20260926 07:20:02.608691 6292 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:02.602491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:51767 (local address 127.28.235.254:39849)
0926 07:20:02.602653 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:02.602657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:02.602811 (+ 154us) server_negotiation.cc:407] Connection header received
0926 07:20:02.602920 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:02.602922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:02.602971 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:02.603047 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:02.603662 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.604206 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.605310 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:02.605533 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:02.605626 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:02.608550 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:51767: BlockingRecv error: recv got EOF from 127.28.235.195:51767 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 07:20:03 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:03 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:03.210958 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:03.205175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:58603)
0926 07:20:03.205471 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:03.205497 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:03.205591 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:03.205900 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:03.205903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:03.206099 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:03.206204 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:03.206210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.207019 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.207022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:03.207632 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:03.207643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.208105 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.208108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:03.208170 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:03.208808 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:03.208820 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:03.210827 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":116,"threads_started":1}
W20260926 07:20:03.211206 6296 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:03.205318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:58603 (local address 127.28.235.254:39849)
0926 07:20:03.205615 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:03.205618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:03.205634 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:20:03.205687 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:03.205690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:03.205732 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:03.205823 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:03.206334 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.206903 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.207756 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.207977 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.208066 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:03.211046 (+ 2980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:58603: BlockingRecv error: recv got EOF from 127.28.235.193:58603 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
Sep 26 07:20:03 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:03 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:03.410751 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:03.404914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:58965)
0926 07:20:03.405258 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:03.405272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:03.405364 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:03.405649 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:03.405651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:03.405830 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:03.405930 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:03.405936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.406777 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.406780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:03.407423 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:03.407435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.407947 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.407950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:03.408019 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:03.408664 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:03.408676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:03.410611 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":146,"threads_started":1}
W20260926 07:20:03.410995 6296 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:03.405140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:58965 (local address 127.28.235.254:39849)
0926 07:20:03.405304 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:03.405307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:03.405323 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:20:03.405470 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:03.405472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:03.405521 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:03.405601 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:03.406054 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.406638 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.407569 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.407793 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.407884 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:03.410831 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:58965: BlockingRecv error: recv got EOF from 127.28.235.194:58965 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 07:20:03 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:03 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:03.622467 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:03.609602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:37557)
0926 07:20:03.609961 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:03.610032 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:03.610195 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:03.610557 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:03.610560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:03.610806 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:03.610957 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:03.610965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.612090 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.612096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:03.612711 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:03.612719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.613304 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.613307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:03.613379 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:03.620075 (+ 6696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:03.620092 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:03.622273 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":76,"threads_started":1}
W20260926 07:20:03.622668 6296 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:03.609867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:37557 (local address 127.28.235.254:39849)
0926 07:20:03.610080 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:03.610084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:03.610124 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:20:03.610302 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:03.610305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:03.610389 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:03.610483 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:03.611100 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.611829 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.612847 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:03.613155 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:03.613262 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:03.622537 (+ 9275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:37557: BlockingRecv error: recv got EOF from 127.28.235.195:37557 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 07:20:04 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:04 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:04.217133 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:04.211773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:41047)
0926 07:20:04.212138 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:04.212152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:04.212245 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:04.212512 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:04.212514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:04.212691 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:04.212791 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:04.212796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.213563 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.213566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:04.214147 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:04.214155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.214559 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.214561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:04.214626 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:04.215122 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:04.215134 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:04.216910 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
W20260926 07:20:04.217252 6300 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:04.211927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:41047 (local address 127.28.235.254:39849)
0926 07:20:04.212179 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:04.212183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:04.212198 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:20:04.212336 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:04.212339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:04.212381 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:04.212465 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:04.212894 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.213459 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.214250 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.214463 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.214537 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:04.217099 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:41047: BlockingRecv error: recv got EOF from 127.28.235.193:41047 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
Sep 26 07:20:04 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:04 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:04.417533 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:04.411721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:55163)
0926 07:20:04.412023 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:04.412087 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:04.412192 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:04.412448 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:04.412450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:04.412636 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:04.412731 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:04.412736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.413491 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.413493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:04.414097 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:04.414106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.414565 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.414568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:04.414646 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:04.415124 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:04.415135 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:04.417340 (+ 2205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":130,"threads_started":1}
W20260926 07:20:04.417614 6300 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:04.411870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:55163 (local address 127.28.235.254:39849)
0926 07:20:04.412020 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:04.412023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:04.412163 (+ 140us) server_negotiation.cc:407] Connection header received
0926 07:20:04.412284 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:04.412286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:04.412334 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:04.412401 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:04.412842 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.413386 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.414204 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.414414 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.414494 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:04.417486 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:55163: BlockingRecv error: recv got EOF from 127.28.235.194:55163 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 07:20:04 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:04 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:04.629565 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:04.623412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:54331)
0926 07:20:04.623760 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:04.623776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:04.623942 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:04.624267 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:04.624270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:04.624519 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:04.624624 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:04.624631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.625605 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.625612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:04.626245 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:04.626253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.626758 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.626763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:04.626858 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:04.627453 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:04.627467 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:04.629436 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
W20260926 07:20:04.629760 6300 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:04.623508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:54331 (local address 127.28.235.254:39849)
0926 07:20:04.623719 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:04.623725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:04.623886 (+ 161us) server_negotiation.cc:407] Connection header received
0926 07:20:04.624073 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:04.624079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:04.624142 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:04.624238 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:04.624761 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.625485 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.626377 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:04.626631 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:04.626753 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:04.629620 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:54331: BlockingRecv error: recv got EOF from 127.28.235.195:54331 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 26 07:20:05 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:05 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:05.224090 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:05.218154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:53933)
0926 07:20:05.218549 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:05.218564 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:05.218699 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:05.219008 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:05.219012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:05.219219 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:05.219313 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:05.219318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.220117 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.220122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:05.220690 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:05.220698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.221240 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.221244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:05.221381 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:05.221963 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:05.221976 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:05.223898 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":109,"threads_started":1}
W20260926 07:20:05.224164 6304 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:05.218229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:53933 (local address 127.28.235.254:39849)
0926 07:20:05.218544 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:05.218548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:05.218646 (+ 98us) server_negotiation.cc:407] Connection header received
0926 07:20:05.218804 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:05.218806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:05.218853 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:05.218967 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:05.219459 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.220000 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.220814 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.221121 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.221233 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:05.224035 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:53933: BlockingRecv error: recv got EOF from 127.28.235.193:53933 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
Sep 26 07:20:05 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:05 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:05.424544 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:05.418655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:35407)
0926 07:20:05.418975 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:05.418989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:05.419090 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:05.419395 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:05.419397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:05.419582 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:05.419679 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:05.419684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.420505 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.420508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:05.421282 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:05.421292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.421742 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.421745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:05.421808 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:05.422382 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:05.422398 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:05.424421 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":93,"threads_started":1}
W20260926 07:20:05.424695 6304 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:05.418745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:35407 (local address 127.28.235.254:39849)
0926 07:20:05.418978 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:05.418981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:05.419099 (+ 118us) server_negotiation.cc:407] Connection header received
0926 07:20:05.419185 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:05.419187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:05.419239 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:05.419316 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:05.419823 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.420372 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.421402 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.421624 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.421709 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:05.424570 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:35407: BlockingRecv error: recv got EOF from 127.28.235.194:35407 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 07:20:05 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:05 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:05.636546 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:05.630604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:32989)
0926 07:20:05.630946 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:05.630958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:05.631099 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:05.631402 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:05.631406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:05.631633 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:05.631729 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:05.631734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.632511 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.632515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:05.633153 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:05.633162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.633620 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.633624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:05.633711 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:05.634317 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:05.634329 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:05.636345 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":80,"threads_started":1}
W20260926 07:20:05.636606 6304 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:05.630679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:32989 (local address 127.28.235.254:39849)
0926 07:20:05.630849 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:05.630853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:05.631060 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:20:05.631216 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:05.631219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:05.631287 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:05.631375 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:05.631850 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.632382 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.633275 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:05.633502 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:05.633613 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:05.636492 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:32989: BlockingRecv error: recv got EOF from 127.28.235.195:32989 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 07:20:06 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:06 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:06.230626 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:06.225075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:36389)
0926 07:20:06.225423 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:06.225436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:06.225582 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:06.225931 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:06.225934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:06.226178 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:06.226287 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:06.226293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.227021 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.227023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:06.227594 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:06.227601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.228083 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.228086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:06.228151 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:06.228626 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:06.228637 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:06.230498 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":96,"threads_started":1}
W20260926 07:20:06.230785 6308 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:06.225135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:36389 (local address 127.28.235.254:39849)
0926 07:20:06.225427 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:06.225431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:06.225543 (+ 112us) server_negotiation.cc:407] Connection header received
0926 07:20:06.225720 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:06.225724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:06.225780 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:06.225889 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:06.226375 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.226925 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.227724 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.227969 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.228059 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:06.230648 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:36389: BlockingRecv error: recv got EOF from 127.28.235.193:36389 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
Sep 26 07:20:06 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:06 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:06.431259 6309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:06.425437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:60145)
0926 07:20:06.425786 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:06.425800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:06.425899 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:06.426146 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:06.426148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:06.426336 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:06.426430 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:06.426435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.427240 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.427243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:06.427860 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:06.427870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.428334 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.428337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:06.428408 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:06.429060 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:06.429072 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:06.431057 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":94,"threads_started":1}
W20260926 07:20:06.431299 6308 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:06.425661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:60145 (local address 127.28.235.254:39849)
0926 07:20:06.425809 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:06.425813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:06.425853 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:20:06.425985 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:06.425987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:06.426031 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:06.426104 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:06.426554 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.427084 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.427987 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.428222 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.428304 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:06.431195 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:60145: BlockingRecv error: recv got EOF from 127.28.235.194:60145 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 07:20:06 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:06 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:06.644575 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:06.637589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:45697)
0926 07:20:06.637839 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:06.637853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:06.637972 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:06.638250 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:06.638272 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:06.638543 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:06.638692 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:06.638700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.639972 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.639975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:06.640809 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:06.640821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.641663 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.641667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:06.641771 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:06.642372 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:06.642388 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:06.644455 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
W20260926 07:20:06.644853 6308 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:06.637630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:45697 (local address 127.28.235.254:39849)
0926 07:20:06.637769 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:06.637774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:06.637918 (+ 144us) server_negotiation.cc:407] Connection header received
0926 07:20:06.638072 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:06.638077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:06.638134 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:06.638219 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:06.638847 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.639627 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.640940 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:06.641541 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:06.641657 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:06.644679 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:45697: BlockingRecv error: recv got EOF from 127.28.235.195:45697 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 07:20:07 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:07 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:07.237540 6311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:07.231574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:38765)
0926 07:20:07.231851 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:07.231865 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:07.231959 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:07.232309 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:07.232312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:07.232542 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:07.232636 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:07.232642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.233495 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.233500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:07.234139 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:07.234148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.234635 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.234638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:07.234723 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:07.235374 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:07.235387 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:07.237340 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
W20260926 07:20:07.237707 6312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:07.231649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:38765 (local address 127.28.235.254:39849)
0926 07:20:07.231907 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:07.231911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:07.231926 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:20:07.232101 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:07.232104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:07.232154 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:07.232287 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:07.232761 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.233378 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.234275 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.234517 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.234630 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:07.237561 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:38765: BlockingRecv error: recv got EOF from 127.28.235.193:38765 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":66,"threads_started":1}
Sep 26 07:20:07 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:07 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:07.437837 6313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:07.432174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:41059)
0926 07:20:07.432443 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:07.432459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:07.432613 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:07.432909 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:07.432912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:07.433212 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:07.433309 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:07.433315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.434108 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.434111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:07.434668 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:07.434675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.435152 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.435155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:07.435240 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:07.435723 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:07.435741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:07.437714 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":83,"threads_started":1}
W20260926 07:20:07.438158 6312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:07.432236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:41059 (local address 127.28.235.254:39849)
0926 07:20:07.432426 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:07.432431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:07.432552 (+ 121us) server_negotiation.cc:407] Connection header received
0926 07:20:07.432719 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:07.432722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:07.432770 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:07.432869 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:07.433429 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.433984 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.434782 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.435025 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.435137 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:07.437851 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:41059: BlockingRecv error: recv got EOF from 127.28.235.194:41059 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 07:20:07 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:07 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:07.651338 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:07.645534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:57389)
0926 07:20:07.645802 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:07.645815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:07.645917 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:07.646182 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:07.646184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:07.646370 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:07.646465 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:07.646470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.647270 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.647272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:07.647998 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:07.648016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.648555 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.648558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:07.648623 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:07.649176 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:07.649189 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:07.651223 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
W20260926 07:20:07.651666 6312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:07.645627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:57389 (local address 127.28.235.254:39849)
0926 07:20:07.645784 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:07.645788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:07.645887 (+ 99us) server_negotiation.cc:407] Connection header received
0926 07:20:07.645996 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:07.645999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:07.646071 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:07.646146 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:07.646575 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.647151 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.648122 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:07.648432 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:07.648540 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:07.651382 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:57389: BlockingRecv error: recv got EOF from 127.28.235.195:57389 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 07:20:08 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:08 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:08.244462 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:08.238354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:32857)
0926 07:20:08.238784 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:08.238798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:08.239020 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:08.239323 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:08.239326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:08.239515 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:08.239616 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:08.239621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.240456 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.240459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:08.241227 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:08.241240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.241700 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.241702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:08.241764 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:08.242304 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:08.242315 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:08.244337 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":121,"threads_started":1}
W20260926 07:20:08.244778 6316 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:08.238541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:32857 (local address 127.28.235.254:39849)
0926 07:20:08.238788 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:08.238791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:08.238970 (+ 179us) server_negotiation.cc:407] Connection header received
0926 07:20:08.239139 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:08.239141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:08.239189 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:08.239275 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:08.239732 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.240325 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.241349 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.241595 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.241676 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:08.244610 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:32857: BlockingRecv error: recv got EOF from 127.28.235.193:32857 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":60,"threads_started":1}
Sep 26 07:20:08 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:08 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:08.444523 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:08.438716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:33447)
0926 07:20:08.439085 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:08.439100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:08.439218 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:08.439454 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:08.439456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:08.439644 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:08.439737 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:08.439742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.440588 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.440591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:08.441321 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:08.441331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.441919 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.441922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:08.442011 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:08.442486 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:08.442496 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:08.444350 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":96,"threads_started":1}
W20260926 07:20:08.444593 6316 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:08.438781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:33447 (local address 127.28.235.254:39849)
0926 07:20:08.438944 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:08.438949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:08.439191 (+ 242us) server_negotiation.cc:407] Connection header received
0926 07:20:08.439302 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:08.439305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:08.439352 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:08.439412 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:08.439858 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.440483 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.441437 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.441797 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.441910 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:08.444485 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:33447: BlockingRecv error: recv got EOF from 127.28.235.194:33447 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 07:20:08 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:08 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:08.673568 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:08.652216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:49425)
0926 07:20:08.652551 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:08.652565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:08.652665 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:08.652899 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:08.652901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:08.653119 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:08.653217 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:08.653223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.653932 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.653935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:08.654481 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:08.654489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.654916 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.654918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:08.654980 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:08.671045 (+ 16065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:08.671070 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:08.673424 (+ 2354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":100,"threads_started":1}
W20260926 07:20:08.673857 6316 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:08.652279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:49425 (local address 127.28.235.254:39849)
0926 07:20:08.652428 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:08.652433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:08.652640 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:20:08.652742 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:08.652744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:08.652791 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:08.652859 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:08.653319 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.653836 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.654602 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:08.654817 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:08.654895 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:08.673591 (+ 18696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:49425: BlockingRecv error: recv got EOF from 127.28.235.195:49425 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 07:20:09 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:09 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:09.251132 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:09.245497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:39363)
0926 07:20:09.245829 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:09.245846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:09.245952 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:09.246219 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:09.246222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:09.246402 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:09.246501 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:09.246506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.247232 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.247234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:09.247808 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:09.247816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.248293 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.248297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:09.248367 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:09.248990 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:09.249003 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:09.250925 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
W20260926 07:20:09.251189 6320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:09.245685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:39363 (local address 127.28.235.254:39849)
0926 07:20:09.245945 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:09.245951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:09.245970 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:20:09.246048 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:09.246051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:09.246096 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:09.246172 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:09.246610 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.247136 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.247917 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.248152 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.248233 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:09.251064 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:39363: BlockingRecv error: recv got EOF from 127.28.235.193:39363 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
Sep 26 07:20:09 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:09 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:09.451485 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:09.445477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:52329)
0926 07:20:09.445773 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:09.445786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:09.445886 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:09.446147 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:09.446149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:09.446385 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:09.446483 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:09.446488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.447361 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.447363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:09.447938 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:09.447945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.448622 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.448629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:09.448724 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:09.449374 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:09.449393 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:09.451364 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":121,"threads_started":1}
W20260926 07:20:09.451756 6320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:09.445614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:52329 (local address 127.28.235.254:39849)
0926 07:20:09.445774 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:09.445778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:09.445859 (+ 81us) server_negotiation.cc:407] Connection header received
0926 07:20:09.445971 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:09.445974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:09.446020 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:09.446107 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:09.446601 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.447265 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.448147 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.448481 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.448591 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:09.451504 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:52329: BlockingRecv error: recv got EOF from 127.28.235.194:52329 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 07:20:09 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:09 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:09.680359 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:09.674435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:59825)
0926 07:20:09.674806 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:09.674819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:09.674920 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:09.675185 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:09.675187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:09.675359 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:09.675471 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:09.675476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.676371 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.676377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:09.677203 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:09.677212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.677699 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.677703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:09.677788 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:09.678309 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:09.678321 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:09.680179 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":126,"threads_started":1}
W20260926 07:20:09.680426 6320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:09.674590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:59825 (local address 127.28.235.254:39849)
0926 07:20:09.674773 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:09.674777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:09.674895 (+ 118us) server_negotiation.cc:407] Connection header received
0926 07:20:09.675013 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:09.675015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:09.675063 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:09.675147 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:09.675604 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.676235 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.677333 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:09.677575 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:09.677689 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:09.680304 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:59825: BlockingRecv error: recv got EOF from 127.28.235.195:59825 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 07:20:10 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:10 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:10.258225 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:10.252211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:58045)
0926 07:20:10.252569 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:10.252590 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:10.252716 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:10.253079 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:10.253082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:10.253315 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:10.253453 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:10.253460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.254301 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.254303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:10.254871 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:10.254878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.255365 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.255368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:10.255428 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:10.256010 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:10.256024 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:10.258045 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":116,"threads_started":1}
W20260926 07:20:10.258287 6324 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:10.252384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:58045 (local address 127.28.235.254:39849)
0926 07:20:10.252601 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:10.252606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:10.252674 (+ 68us) server_negotiation.cc:407] Connection header received
0926 07:20:10.252831 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:10.252834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:10.252893 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:10.252996 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:10.253635 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.254201 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.254998 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.255260 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.255348 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:10.258164 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:58045: BlockingRecv error: recv got EOF from 127.28.235.193:58045 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
Sep 26 07:20:10 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:10 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:10.458213 6325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:10.452360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:44055)
0926 07:20:10.452623 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:10.452645 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:10.452744 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:10.452975 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:10.452978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:10.453211 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:10.453307 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:10.453313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.454356 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.454362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:10.454933 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:10.454942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.455423 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.455426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:10.455496 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:10.456101 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:10.456113 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:10.458018 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
W20260926 07:20:10.458247 6324 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:10.452415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:44055 (local address 127.28.235.254:39849)
0926 07:20:10.452560 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:10.452564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:10.452714 (+ 150us) server_negotiation.cc:407] Connection header received
0926 07:20:10.452810 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:10.452812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:10.452856 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:10.452953 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:10.453425 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.454216 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.455041 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.455256 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.455334 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:10.458145 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:44055: BlockingRecv error: recv got EOF from 127.28.235.194:44055 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 07:20:10 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:10 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:10.686803 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:10.681234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:42489)
0926 07:20:10.681498 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:10.681510 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:10.681613 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:10.681847 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:10.681849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:10.682022 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:10.682118 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:10.682124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.682836 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.682838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:10.683379 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:10.683386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.683801 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.683804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:10.683862 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:10.684364 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:10.684375 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:10.686605 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":104,"threads_started":1}
W20260926 07:20:10.686860 6324 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:10.681423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:42489 (local address 127.28.235.254:39849)
0926 07:20:10.681550 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:10.681556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:10.681574 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:20:10.681696 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:10.681699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:10.681744 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:10.681813 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:10.682219 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.682745 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.683483 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:10.683697 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:10.683776 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:10.686749 (+ 2973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:42489: BlockingRecv error: recv got EOF from 127.28.235.195:42489 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 07:20:11 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:11 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:11.265283 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:11.259249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:49627)
0926 07:20:11.259539 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:11.259553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:11.259643 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:11.259929 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:11.259932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:11.260162 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:11.260258 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:11.260265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.261243 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.261247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:11.261997 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:11.262008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.262460 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.262462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:11.262524 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:11.263022 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:11.263034 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:11.265155 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
W20260926 07:20:11.265429 6328 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:11.259320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:49627 (local address 127.28.235.254:39849)
0926 07:20:11.259588 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:11.259592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:11.259607 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:20:11.259739 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:11.259741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:11.259784 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:11.259867 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:11.260378 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.261112 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.262115 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.262340 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.262420 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:11.265290 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:49627: BlockingRecv error: recv got EOF from 127.28.235.193:49627 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
Sep 26 07:20:11 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:11 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:11.465160 6329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:11.459122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:48693)
0926 07:20:11.459393 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:11.459407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:11.459503 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:11.459807 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:11.459810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:11.460029 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:11.460129 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:11.460136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.460898 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.460901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:11.461606 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:11.461617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.462101 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.462105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:11.462173 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:11.462687 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:11.462701 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:11.464931 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
W20260926 07:20:11.465199 6328 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:11.459170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:48693 (local address 127.28.235.254:39849)
0926 07:20:11.459341 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:11.459345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:11.459510 (+ 165us) server_negotiation.cc:407] Connection header received
0926 07:20:11.459582 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:11.459585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:11.459632 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:11.459704 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:11.460250 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.460780 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.461739 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.461961 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.462074 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:11.465098 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:48693: BlockingRecv error: recv got EOF from 127.28.235.194:48693 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 07:20:11 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:11 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:11.693672 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:11.687874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:35769)
0926 07:20:11.688179 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:11.688193 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:11.688292 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:11.688561 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:11.688563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:11.688733 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:11.688830 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:11.688836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.689781 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.689783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:11.690354 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:11.690362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.690812 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.690814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:11.690876 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:11.691391 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:11.691402 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:11.693484 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
W20260926 07:20:11.693760 6328 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:11.688049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:35769 (local address 127.28.235.254:39849)
0926 07:20:11.688183 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:11.688186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:11.688286 (+ 100us) server_negotiation.cc:407] Connection header received
0926 07:20:11.688363 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:11.688366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:11.688412 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:11.688488 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:11.688979 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.689681 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.690479 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:11.690712 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:11.690793 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:11.693640 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:35769: BlockingRecv error: recv got EOF from 127.28.235.195:35769 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 07:20:12 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:12 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:12.271498 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:12.266192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:49773)
0926 07:20:12.266506 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:12.266536 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:12.266637 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:12.266895 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:12.266897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:12.267084 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:12.267184 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:12.267190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.267937 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.267940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:12.268490 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:12.268497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.268942 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.268944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:12.269005 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:12.269591 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:12.269603 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:12.271371 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":85,"threads_started":1}
W20260926 07:20:12.271641 6332 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:12.266270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:49773 (local address 127.28.235.254:39849)
0926 07:20:12.266527 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:12.266530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:12.266607 (+ 77us) server_negotiation.cc:407] Connection header received
0926 07:20:12.266729 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:12.266732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:12.266778 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:12.266856 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:12.267307 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.267843 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.268616 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.268840 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.268920 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:12.271506 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:49773: BlockingRecv error: recv got EOF from 127.28.235.193:49773 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":110,"threads_started":1}
Sep 26 07:20:12 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:12 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:12.472155 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:12.466198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:58743)
0926 07:20:12.466536 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:12.466549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:12.466650 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:12.466994 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:12.466997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:12.467180 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:12.467275 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:12.467281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.468083 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.468086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:12.468659 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:12.468668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.469235 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.469238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:12.469315 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:12.469924 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:12.469936 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:12.472012 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":125,"threads_started":1}
W20260926 07:20:12.472359 6332 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:12.466235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:58743 (local address 127.28.235.254:39849)
0926 07:20:12.466375 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:12.466378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:12.466615 (+ 237us) server_negotiation.cc:407] Connection header received
0926 07:20:12.466801 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:12.466803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:12.466863 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:12.466958 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:12.467379 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.467971 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.468780 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.469100 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.469201 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:12.472184 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:58743: BlockingRecv error: recv got EOF from 127.28.235.194:58743 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 07:20:12 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:12 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:12.701053 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:12.694555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:42919)
0926 07:20:12.694906 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:12.694921 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:12.695053 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:12.695297 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:12.695299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:12.695511 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:12.695616 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:12.695623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.696552 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.696555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:12.697293 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:12.697304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.697875 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.697879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:12.698035 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:12.698753 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:12.698791 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:12.700811 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":73,"threads_started":1}
W20260926 07:20:12.701076 6332 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:12.694723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:42919 (local address 127.28.235.254:39849)
0926 07:20:12.694876 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:12.694912 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:12.695019 (+ 107us) server_negotiation.cc:407] Connection header received
0926 07:20:12.695139 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:12.695141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:12.695189 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:12.695256 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:12.695724 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.696433 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.697419 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:12.697651 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:12.697734 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:12.700942 (+ 3208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:42919: BlockingRecv error: recv got EOF from 127.28.235.195:42919 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 07:20:13 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:13 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:13.278285 6335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:13.272505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:49537)
0926 07:20:13.272799 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:13.272812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:13.272901 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:13.273290 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:13.273293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:13.273484 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:13.273586 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:13.273592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.274406 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.274409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:13.275024 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:13.275036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.275477 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.275480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:13.275546 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:13.276050 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:13.276062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:13.278122 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":119,"threads_started":1}
W20260926 07:20:13.278611 6336 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:13.272602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:49537 (local address 127.28.235.254:39849)
0926 07:20:13.272851 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:13.272854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:13.272868 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:20:13.273042 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:13.273044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:13.273119 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:13.273219 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:13.273728 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.274289 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.275140 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.275368 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.275450 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:13.278370 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:49537: BlockingRecv error: recv got EOF from 127.28.235.193:49537 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
Sep 26 07:20:13 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:13 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:13.479300 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:13.473212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:46817)
0926 07:20:13.473512 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:13.473525 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:13.473676 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:13.473965 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:13.473968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:13.474210 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:13.474333 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:13.474339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.475253 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.475258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:13.475910 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:13.475919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.476387 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.476390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:13.476470 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:13.477114 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:13.477127 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:13.479180 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
W20260926 07:20:13.479621 6336 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:13.473262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:46817 (local address 127.28.235.254:39849)
0926 07:20:13.473456 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:13.473461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:13.473614 (+ 153us) server_negotiation.cc:407] Connection header received
0926 07:20:13.473787 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:13.473791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:13.473845 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:13.473935 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:13.474492 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.475147 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.476039 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.476270 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.476381 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:13.479408 (+ 3027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:46817: BlockingRecv error: recv got EOF from 127.28.235.194:46817 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 07:20:13 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:13 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:13.712075 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:13.701903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:44231)
0926 07:20:13.702298 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:13.702310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:13.702446 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:13.702754 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:13.702757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:13.702987 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:13.703088 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:13.703094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.703861 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.703865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:13.704448 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:13.704456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.704928 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.704931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:13.705052 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:13.709874 (+ 4822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:13.709915 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:13.711888 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":125,"threads_started":1}
W20260926 07:20:13.712193 6336 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:13.702023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:44231 (local address 127.28.235.254:39849)
0926 07:20:13.702158 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:13.702163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:13.702390 (+ 227us) server_negotiation.cc:407] Connection header received
0926 07:20:13.702572 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:13.702576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:13.702635 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:13.702725 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:13.703209 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.703743 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.704581 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:13.704811 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:13.704922 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:13.712071 (+ 7149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:44231: BlockingRecv error: recv got EOF from 127.28.235.195:44231 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 07:20:14 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:14 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:14.285560 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:14.279245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:39771)
0926 07:20:14.279615 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:14.279630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:14.279728 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:14.280024 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:14.280026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:14.280212 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:14.280309 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:14.280315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.281389 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.281392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:14.282181 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:14.282191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.282756 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.282759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:14.282843 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:14.283419 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:14.283432 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:14.285390 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
W20260926 07:20:14.285715 6340 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:14.279388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:39771 (local address 127.28.235.254:39849)
0926 07:20:14.279658 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:14.279661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:14.279681 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:20:14.279828 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:14.279831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:14.279886 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:14.279973 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:14.280440 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.281264 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.282306 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.282634 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.282750 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:14.285548 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:39771: BlockingRecv error: recv got EOF from 127.28.235.193:39771 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":65,"threads_started":1}
Sep 26 07:20:14 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:14 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:14.486289 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:14.480279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:48837)
0926 07:20:14.480538 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:14.480564 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:14.480679 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:14.480944 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:14.480946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:14.481189 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:14.481287 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:14.481293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.482142 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.482145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:14.482757 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:14.482766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.483257 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.483260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:14.483324 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:14.483953 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:14.483965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:14.486092 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
W20260926 07:20:14.486423 6340 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:14.480443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:48837 (local address 127.28.235.254:39849)
0926 07:20:14.480625 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:14.480628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:14.480644 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:20:14.480773 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:14.480775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:14.480819 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:14.480914 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:14.481424 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.481993 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.482902 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.483139 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.483247 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:14.486281 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:48837: BlockingRecv error: recv got EOF from 127.28.235.194:48837 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 26 07:20:14 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:14 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:14.718357 6342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:14.712901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:57511)
0926 07:20:14.713261 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:14.713273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:14.713371 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:14.713611 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:14.713614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:14.713798 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:14.713906 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:14.713912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.714644 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.714646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:14.715207 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:14.715215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.715651 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.715654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:14.715717 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:14.716217 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:14.716228 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:14.718180 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
W20260926 07:20:14.718423 6340 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:14.713052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:57511 (local address 127.28.235.254:39849)
0926 07:20:14.713205 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:14.713209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:14.713346 (+ 137us) server_negotiation.cc:407] Connection header received
0926 07:20:14.713452 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:14.713455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:14.713504 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:14.713575 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:14.714014 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.714540 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.715330 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:14.715553 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:14.715639 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:14.718310 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:57511: BlockingRecv error: recv got EOF from 127.28.235.195:57511 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 07:20:15 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:15 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:15.293128 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:15.286465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:49625)
0926 07:20:15.286772 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:15.286834 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:15.286951 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:15.287340 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:15.287343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:15.287537 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:15.287641 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:15.287647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.288469 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.288472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:15.289273 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:15.289285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.289925 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.289928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:15.290045 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:15.290609 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:15.290622 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:15.292859 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":129,"threads_started":1}
W20260926 07:20:15.293377 6344 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:15.286539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:49625 (local address 127.28.235.254:39849)
0926 07:20:15.286901 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:15.286905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:15.286920 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:20:15.287068 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:15.287070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:15.287117 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:15.287203 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:15.287771 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.288326 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.289409 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.289782 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.289891 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:15.293215 (+ 3324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:49625: BlockingRecv error: recv got EOF from 127.28.235.193:49625 (error 108)
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":93,"threads_started":1}
Sep 26 07:20:15 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:15 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:15.493294 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:15.487293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:43773)
0926 07:20:15.487602 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:15.487616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:15.487721 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:15.488018 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:15.488021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:15.488230 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:15.488330 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:15.488374 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.489191 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.489194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:15.489796 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:15.489806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.490298 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.490301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:15.490371 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:15.491026 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:15.491043 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:15.493166 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
W20260926 07:20:15.493505 6344 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:15.487345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:43773 (local address 127.28.235.254:39849)
0926 07:20:15.487499 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:15.487503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:15.487688 (+ 185us) server_negotiation.cc:407] Connection header received
0926 07:20:15.487804 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:15.487806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:15.487853 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:15.487934 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:15.488489 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.489084 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.489902 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.490185 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.490267 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:15.493353 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:43773: BlockingRecv error: recv got EOF from 127.28.235.194:43773 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 07:20:15 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:15 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.195@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:15.724929 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:15.719232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.195:56649)
0926 07:20:15.719494 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:15.719507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:15.719615 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:15.719840 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:15.719843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:15.720039 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:15.720136 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:15.720141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.720885 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.720888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:15.721602 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:15.721611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.722080 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.722082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:15.722147 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:15.722770 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:15.722783 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:15.724794 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
W20260926 07:20:15.725235 6344 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:15.719379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:56649 (local address 127.28.235.254:39849)
0926 07:20:15.719507 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:15.719511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:15.719574 (+ 63us) server_negotiation.cc:407] Connection header received
0926 07:20:15.719685 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:15.719687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:15.719731 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:15.719795 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:15.720258 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.720789 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.721728 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:15.721951 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:15.722048 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:15.725086 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.195:56649: BlockingRecv error: recv got EOF from 127.28.235.195:56649 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 07:20:16 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:16 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.193@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:16.301537 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:16.294214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.193:36103)
0926 07:20:16.294516 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:16.294530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:16.294618 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:16.295036 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:16.295049 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:16.295236 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:16.295342 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:16.295348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:16.296389 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:16.296392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:16.297484 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:16.297496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:16.298108 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:16.298111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:16.298182 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:16.298995 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:16.299008 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:16.301407 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":115,"threads_started":1}
W20260926 07:20:16.301772 6348 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:16.294324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:36103 (local address 127.28.235.254:39849)
0926 07:20:16.294587 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:16.294591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:16.294606 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:20:16.294799 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:16.294801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:16.294848 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:16.294950 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:16.295498 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:16.296249 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:16.297690 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:16.297936 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:16.298076 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:16.301607 (+ 3531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.193:36103: BlockingRecv error: recv got EOF from 127.28.235.193:36103 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
Sep 26 07:20:16 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:20:16 dist-test-slave-7zlw krb5kdc[5289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.235.194@KRBTEST.COM for oryx/127.28.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:20:16.500496 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:16.494465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39849 (local address 127.28.235.194:52729)
0926 07:20:16.494741 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:16.494756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:16.494864 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:16.495207 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:16.495209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:16.495396 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:16.495493 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:16.495500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:16.496519 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:16.496524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:16.497326 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:16.497336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:16.497814 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:16.497817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:16.497984 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:16.498514 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:16.498525 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:16.500374 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.235.254:39849: Server oryx/127.28.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":128,"threads_started":1}
W20260926 07:20:16.500746 6348 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:16.494579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:52729 (local address 127.28.235.254:39849)
0926 07:20:16.494730 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:16.494733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:16.494837 (+ 104us) server_negotiation.cc:407] Connection header received
0926 07:20:16.494995 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:16.494998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:16.495047 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:16.495137 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:16.495647 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:16.496382 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:16.497446 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:16.497694 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:16.497812 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:16.500604 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.235.194:52729: BlockingRecv error: recv got EOF from 127.28.235.194:52729 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:20:16.592481 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 5872
I20260926 07:20:16.598707 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 6025
I20260926 07:20:16.604760 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 6157
I20260926 07:20:16.610273 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 5802
2026-09-26T07:20:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18160 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T07:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:20:16Z Disabled control of system clock
I20260926 07:20:16.641937 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:45869
--webserver_interface=127.28.235.254
--webserver_port=0
--builtin_ntp_servers=127.28.235.212:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:45869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:20:16.771997 6353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:16.772323 6353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:16.772360 6353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:16.776990 6353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:20:16.777204 6353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:16.777236 6353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:16.777302 6353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:20:16.777328 6353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:20:16.782507 6353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:45869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:45869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:16.783739 6353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:16.785229 6353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:16.795037 6361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:16.795037 6359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:16.795137 6353 server_base.cc:1034] running on GCE node
W20260926 07:20:16.795037 6358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:16.796082 6353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:16.797592 6353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:16.798821 6353 hybrid_clock.cc:648] HybridClock initialized: now 1790407216798778 us; error 52 us; skew 500 ppm
I20260926 07:20:16.800935 6353 webserver.cc:460] Webserver started at http://127.28.235.254:43163/ using document root <none> and password file <none>
I20260926 07:20:16.801630 6353 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:16.801729 6353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:16.801978 6353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:16.804509 6353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "e7905207b84544cda93adc99348f6351"
format_stamp: "Formatted at 2026-09-26 07:20:16 on dist-test-slave-7zlw"
server_key: "016d0ad6efb447c64ae8e18146e76cee"
server_key_iv: "319c6f27bb887b5f225339e377350dcf"
server_key_version: "encryptionkey@0"
I20260926 07:20:16.805217 6353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "e7905207b84544cda93adc99348f6351"
format_stamp: "Formatted at 2026-09-26 07:20:16 on dist-test-slave-7zlw"
server_key: "016d0ad6efb447c64ae8e18146e76cee"
server_key_iv: "319c6f27bb887b5f225339e377350dcf"
server_key_version: "encryptionkey@0"
I20260926 07:20:16.809422 6353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:20:16.812047 6367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:16.813339 6353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:20:16.813508 6353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "e7905207b84544cda93adc99348f6351"
format_stamp: "Formatted at 2026-09-26 07:20:16 on dist-test-slave-7zlw"
server_key: "016d0ad6efb447c64ae8e18146e76cee"
server_key_iv: "319c6f27bb887b5f225339e377350dcf"
server_key_version: "encryptionkey@0"
I20260926 07:20:16.813644 6353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:16.829449 6353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:16.830245 6353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:16.830466 6353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:16.838066 6353 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:45869
I20260926 07:20:16.838084 6419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:45869 every 8 connection(s)
I20260926 07:20:16.839147 6353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:20:16.843060 6420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:20:16.848675 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 6353
I20260926 07:20:16.848789 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:20:16.849148 29615 external_mini_cluster.cc:1428] Setting key 2b4720fcc59e6dec60c2cbab6ccd46c4
I20260926 07:20:16.849763 6420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351: Bootstrap starting.
I20260926 07:20:16.852550 6420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351: Neither blocks nor log segments found. Creating new log.
I20260926 07:20:16.853463 6420 log.cc:824] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351: Log is configured to *not* fsync() on all Append() calls
I20260926 07:20:16.855612 6420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351: No bootstrap required, opened a new log
I20260926 07:20:16.859148 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:16.850793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.28.235.254:45869)
0926 07:20:16.851533 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:16.851554 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:16.851597 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:20:16.852330 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:16.852350 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:16.852866 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:16.853300 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:16.853770 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:16.854769 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:16.855681 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:16.856227 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:16.856411 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:16.857358 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:16.857369 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:16.857383 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:20:16.857793 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:16.857848 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:20:16.857868 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:16.857981 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:16.858129 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 07:20:16.858399 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":152,"threads_started":1}
I20260926 07:20:16.859669 6420 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7905207b84544cda93adc99348f6351" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45869 } }
I20260926 07:20:16.859947 6420 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:20:16.860045 6420 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7905207b84544cda93adc99348f6351, State: Initialized, Role: FOLLOWER
I20260926 07:20:16.860586 6420 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7905207b84544cda93adc99348f6351" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45869 } }
I20260926 07:20:16.860764 6420 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:20:16.860848 6420 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:20:16.860954 6420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:20:16.862200 6420 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7905207b84544cda93adc99348f6351" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45869 } }
I20260926 07:20:16.862569 6420 leader_election.cc:302] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7905207b84544cda93adc99348f6351; no voters:
I20260926 07:20:16.862915 6420 leader_election.cc:288] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:20:16.863063 6425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:20:16.863328 6425 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [term 1 LEADER]: Becoming Leader. State: Replica: e7905207b84544cda93adc99348f6351, State: Running, Role: LEADER
I20260926 07:20:16.863710 6425 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7905207b84544cda93adc99348f6351" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45869 } }
I20260926 07:20:16.864284 6420 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:20:16.865621 6427 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7905207b84544cda93adc99348f6351. Latest consensus state: current_term: 1 leader_uuid: "e7905207b84544cda93adc99348f6351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7905207b84544cda93adc99348f6351" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45869 } } }
I20260926 07:20:16.865777 6427 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [sys.catalog]: This master's current role is: LEADER
I20260926 07:20:16.866190 6434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:20:16.867456 6426 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7905207b84544cda93adc99348f6351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7905207b84544cda93adc99348f6351" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45869 } } }
I20260926 07:20:16.867624 6426 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351 [sys.catalog]: This master's current role is: LEADER
I20260926 07:20:16.869875 6434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:20:16.876257 6434 catalog_manager.cc:1347] Generated new cluster ID: d97314b5ec744bb69f7e5543fab806f6
I20260926 07:20:16.876366 6434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:20:16.884321 6434 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:20:16.885449 6434 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:20:16.894505 6434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7905207b84544cda93adc99348f6351: Generated new TSK 0
I20260926 07:20:16.895444 6434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:20:16.907634 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45869
--builtin_ntp_servers=127.28.235.212:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 07:20:17.033318 6444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:17.033656 6444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:17.033798 6444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:17.038125 6444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:17.038229 6444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:17.038321 6444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:20:17.043601 6444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:17.044817 6444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:17.046331 6444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:17.056936 6452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:17.056936 6449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:17.056947 6450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:17.057199 6444 server_base.cc:1034] running on GCE node
I20260926 07:20:17.057749 6444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:17.058495 6444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:17.059664 6444 hybrid_clock.cc:648] HybridClock initialized: now 1790407217059643 us; error 29 us; skew 500 ppm
I20260926 07:20:17.061896 6444 webserver.cc:460] Webserver started at http://127.28.235.193:44827/ using document root <none> and password file <none>
I20260926 07:20:17.062529 6444 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:17.062625 6444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:17.062872 6444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:17.065469 6444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "4a9251150d2e4b72bfc7fcdc446787af"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "42040c7ab18204538df1f462d6d3b2cd"
server_key_iv: "092811ce016561d9f1752759dabb945c"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.066134 6444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "4a9251150d2e4b72bfc7fcdc446787af"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "42040c7ab18204538df1f462d6d3b2cd"
server_key_iv: "092811ce016561d9f1752759dabb945c"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.070556 6444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:20:17.073923 6458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:17.075480 6444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:20:17.075666 6444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "4a9251150d2e4b72bfc7fcdc446787af"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "42040c7ab18204538df1f462d6d3b2cd"
server_key_iv: "092811ce016561d9f1752759dabb945c"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.075822 6444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:17.094532 6444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:17.095326 6444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:17.095543 6444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:17.096186 6444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:17.097198 6444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:17.097270 6444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:17.097345 6444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:17.097395 6444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:17.106748 6444 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:33229
I20260926 07:20:17.106768 6571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:33229 every 8 connection(s)
I20260926 07:20:17.107721 6444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:20:17.111069 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 6444
I20260926 07:20:17.111236 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:20:17.111620 29615 external_mini_cluster.cc:1428] Setting key 682e26509ba82e79a7dbde48fcf998e7
I20260926 07:20:17.113981 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45869
--builtin_ntp_servers=127.28.235.212:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:20:17.118491 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:17.109742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:54637 (local address 127.28.235.254:45869)
0926 07:20:17.109883 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:17.109886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:17.111223 (+ 1337us) server_negotiation.cc:407] Connection header received
0926 07:20:17.112885 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:17.112888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:17.112935 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:17.113084 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:17.114140 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.114724 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.115933 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.116199 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.116316 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:17.117700 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:17.117707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:17.117713 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:20:17.117733 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:17.117761 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:20:17.117774 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:17.117864 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:17.118152 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 07:20:17.118228 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:20:17.119805 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:17.110124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:45869 (local address 127.28.235.193:54637)
0926 07:20:17.110852 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:17.110916 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:17.112643 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:17.113297 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:17.113307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:17.113422 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:17.113964 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:17.113977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.114868 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.114879 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:17.115802 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:17.115812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.116374 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.116378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:17.116554 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:17.117361 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:20:17.117364 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:17.117415 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:20:17.117493 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:17.117910 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:17.117923 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:17.117932 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:20:17.118069 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 07:20:17.118442 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260926 07:20:17.121393 6572 heartbeater.cc:347] Connected to a master server at 127.28.235.254:45869
I20260926 07:20:17.121666 6572 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:17.122301 6572 heartbeater.cc:511] Master 127.28.235.254:45869 requested a full tablet report, sending...
I20260926 07:20:17.123981 6384 ts_manager.cc:194] Registered new tserver with Master: 4a9251150d2e4b72bfc7fcdc446787af (127.28.235.193:33229)
I20260926 07:20:17.125296 6384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.193:54637
W20260926 07:20:17.240410 6575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:17.240785 6575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:17.240885 6575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:17.245275 6575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:17.245358 6575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:17.245417 6575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:20:17.250481 6575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:17.251782 6575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:17.253484 6575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:17.264084 6581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:17.264139 6580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:17.264091 6583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:17.264128 6575 server_base.cc:1034] running on GCE node
I20260926 07:20:17.264806 6575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:17.265789 6575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:17.267030 6575 hybrid_clock.cc:648] HybridClock initialized: now 1790407217266998 us; error 53 us; skew 500 ppm
I20260926 07:20:17.269289 6575 webserver.cc:460] Webserver started at http://127.28.235.194:34481/ using document root <none> and password file <none>
I20260926 07:20:17.269955 6575 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:17.270056 6575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:17.270299 6575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:17.272545 6575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "9dd8fc1f3b554fa38cbd74c1a5b652de"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "2e64e449dcb09822f01722d29cfc9163"
server_key_iv: "963b619689ef20c10849177f80c7af1b"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.273269 6575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "9dd8fc1f3b554fa38cbd74c1a5b652de"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "2e64e449dcb09822f01722d29cfc9163"
server_key_iv: "963b619689ef20c10849177f80c7af1b"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.277626 6575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:20:17.280616 6589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:17.282164 6575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:20:17.282315 6575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "9dd8fc1f3b554fa38cbd74c1a5b652de"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "2e64e449dcb09822f01722d29cfc9163"
server_key_iv: "963b619689ef20c10849177f80c7af1b"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.282444 6575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:17.299693 6575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:17.300465 6575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:17.300694 6575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:17.301484 6575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:17.302548 6575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:17.302623 6575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:17.302695 6575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:17.302750 6575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:17.311915 6575 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:44215
I20260926 07:20:17.311939 6702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:44215 every 8 connection(s)
I20260926 07:20:17.312911 6575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:20:17.322647 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 6575
I20260926 07:20:17.322795 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:20:17.323206 29615 external_mini_cluster.cc:1428] Setting key 044ece63f69ab208da3d08f8b6d6bb49
I20260926 07:20:17.324230 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:17.315304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:54667 (local address 127.28.235.254:45869)
0926 07:20:17.315486 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:17.315490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:17.316705 (+ 1215us) server_negotiation.cc:407] Connection header received
0926 07:20:17.318383 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:17.318386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:17.318442 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:17.318591 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:17.319666 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.320276 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.321733 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.322020 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.322117 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:17.323608 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:17.323611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:17.323613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:20:17.323628 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:17.323651 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:20:17.323660 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:17.323729 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:17.324011 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 07:20:17.324090 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 07:20:17.325510 6705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:17.315611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:45869 (local address 127.28.235.194:54667)
0926 07:20:17.316522 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:17.316568 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:17.318061 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:17.318787 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:17.318794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:17.318941 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:17.319482 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:17.319496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.320438 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.320441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:17.321578 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:17.321589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.322203 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.322206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:17.322464 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:17.323217 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:20:17.323222 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:17.323291 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:20:17.323391 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:17.323764 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:17.323781 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:17.323786 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:20:17.323905 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:20:17.324248 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":132,"threads_started":1}
I20260926 07:20:17.325744 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45869
--builtin_ntp_servers=127.28.235.212:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:20:17.326915 6703 heartbeater.cc:347] Connected to a master server at 127.28.235.254:45869
I20260926 07:20:17.327267 6703 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:17.328033 6703 heartbeater.cc:511] Master 127.28.235.254:45869 requested a full tablet report, sending...
I20260926 07:20:17.329512 6384 ts_manager.cc:194] Registered new tserver with Master: 9dd8fc1f3b554fa38cbd74c1a5b652de (127.28.235.194:44215)
I20260926 07:20:17.330261 6384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.194:54667
W20260926 07:20:17.449613 6706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:17.449987 6706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:17.450062 6706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:17.454947 6706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:17.455060 6706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:17.455159 6706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:20:17.460325 6706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:17.461603 6706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:17.463198 6706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:17.473752 6712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:17.473755 6714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:17.473752 6711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:17.474349 6706 server_base.cc:1034] running on GCE node
I20260926 07:20:17.474814 6706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:17.475574 6706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:17.476838 6706 hybrid_clock.cc:648] HybridClock initialized: now 1790407217476816 us; error 41 us; skew 500 ppm
I20260926 07:20:17.479202 6706 webserver.cc:460] Webserver started at http://127.28.235.195:39677/ using document root <none> and password file <none>
I20260926 07:20:17.479878 6706 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:17.479987 6706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:17.480252 6706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:17.482684 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "28a677af35e446f2841a1d526eba09db"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "c5a6a9f5ad20e2a0040d51f9a975268d"
server_key_iv: "2ae1980b636c514fcf25c3a92c509e9e"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.483352 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "28a677af35e446f2841a1d526eba09db"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "c5a6a9f5ad20e2a0040d51f9a975268d"
server_key_iv: "2ae1980b636c514fcf25c3a92c509e9e"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.487936 6706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 07:20:17.490869 6720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:17.492473 6706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:20:17.492643 6706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "28a677af35e446f2841a1d526eba09db"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "c5a6a9f5ad20e2a0040d51f9a975268d"
server_key_iv: "2ae1980b636c514fcf25c3a92c509e9e"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.492779 6706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:17.518903 6706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:17.519690 6706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:17.519912 6706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:17.520568 6706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:17.521598 6706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:17.521672 6706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:17.521744 6706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:17.521792 6706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:17.531165 6706 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:33561
I20260926 07:20:17.531184 6833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:33561 every 8 connection(s)
I20260926 07:20:17.532158 6706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:20:17.534343 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 6706
I20260926 07:20:17.534457 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:20:17.534744 29615 external_mini_cluster.cc:1428] Setting key ef8c83df870ac88a2e277bd3835f0ca7
I20260926 07:20:17.542451 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:17.534339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:41807 (local address 127.28.235.254:45869)
0926 07:20:17.534491 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:17.534495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:17.536021 (+ 1526us) server_negotiation.cc:407] Connection header received
0926 07:20:17.537760 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:17.537762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:17.537807 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:17.537877 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:17.538793 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.539308 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.540529 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.540745 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.540826 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:17.541938 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:17.541941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:17.541943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:20:17.541956 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:17.541976 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:20:17.541984 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:17.542112 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:17.542199 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 07:20:17.542267 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:20:17.543286 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:17.534944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:45869 (local address 127.28.235.195:41807)
0926 07:20:17.535848 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:17.535909 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:17.537561 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:17.538042 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:17.538048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:17.538135 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:17.538666 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:17.538680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.539434 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.539436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:17.540409 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:17.540419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:17.540882 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:17.540885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:17.541069 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:17.541668 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:20:17.541670 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:17.541712 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:20:17.541781 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:17.542066 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:17.542072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:17.542077 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:20:17.542177 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:20:17.542415 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":134,"threads_started":1}
I20260926 07:20:17.544528 6834 heartbeater.cc:347] Connected to a master server at 127.28.235.254:45869
I20260926 07:20:17.544804 6834 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:17.545480 6834 heartbeater.cc:511] Master 127.28.235.254:45869 requested a full tablet report, sending...
I20260926 07:20:17.546645 6384 ts_manager.cc:194] Registered new tserver with Master: 28a677af35e446f2841a1d526eba09db (127.28.235.195:33561)
I20260926 07:20:17.547251 6384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.195:41807
I20260926 07:20:17.549386 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:20:17.552263 6423 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:17.551395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.28.235.254:45869)
0926 07:20:17.551541 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:17.551545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:17.551723 (+ 178us) server_negotiation.cc:407] Connection header received
0926 07:20:17.551811 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:17.551813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:17.551852 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:17.551933 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:17.552156 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60380: BlockingRecv error: recv got EOF from 127.0.0.1:60380 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:20:17.552527 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 6444
I20260926 07:20:17.558691 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 6575
I20260926 07:20:17.564582 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 6706
I20260926 07:20:17.570410 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 6353
2026-09-26T07:20:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (958 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T07:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:20:17Z Disabled control of system clock
I20260926 07:20:17.601414 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43513
--webserver_interface=127.28.235.254
--webserver_port=0
--builtin_ntp_servers=127.28.235.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:43513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:20:17.719491 6845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:17.719807 6845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:17.719848 6845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:17.724102 6845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:20:17.724186 6845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:17.724203 6845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:17.724221 6845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:20:17.724237 6845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:20:17.729594 6845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:43513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:17.730764 6845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:17.732246 6845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:17.741608 6850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:17.741717 6845 server_base.cc:1034] running on GCE node
W20260926 07:20:17.741604 6853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:17.741597 6851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:17.742607 6845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:17.743820 6845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:17.745060 6845 hybrid_clock.cc:648] HybridClock initialized: now 1790407217744969 us; error 54 us; skew 500 ppm
I20260926 07:20:17.747311 6845 webserver.cc:460] Webserver started at http://127.28.235.254:36109/ using document root <none> and password file <none>
I20260926 07:20:17.748037 6845 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:17.748111 6845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:17.748359 6845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:17.750761 6845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "6ea77e6deb5d7fd1d2737e1f93567af6"
server_key_iv: "bdc921894534fb4c6aaf71c5f0cdb49c"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.751418 6845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "6ea77e6deb5d7fd1d2737e1f93567af6"
server_key_iv: "bdc921894534fb4c6aaf71c5f0cdb49c"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.755739 6845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:20:17.758602 6859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:17.760213 6845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:20:17.760399 6845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7"
format_stamp: "Formatted at 2026-09-26 07:20:17 on dist-test-slave-7zlw"
server_key: "6ea77e6deb5d7fd1d2737e1f93567af6"
server_key_iv: "bdc921894534fb4c6aaf71c5f0cdb49c"
server_key_version: "encryptionkey@0"
I20260926 07:20:17.760546 6845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:17.765515 6845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:17.766206 6845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:17.766410 6845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:17.773620 6845 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:43513
I20260926 07:20:17.773624 6911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:43513 every 8 connection(s)
I20260926 07:20:17.774638 6845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:20:17.778083 6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:20:17.778343 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 6845
I20260926 07:20:17.778472 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:20:17.778776 29615 external_mini_cluster.cc:1428] Setting key 448d5447c17755fbf8595435b97c50dc
I20260926 07:20:17.784683 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:17.780238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.28.235.254:43513)
0926 07:20:17.780943 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:17.780958 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:17.780996 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:20:17.781740 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:17.781760 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:17.782176 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:17.782579 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:17.783396 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:17.783401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:17.783413 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:20:17.783423 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:17.783478 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:20:17.783501 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:17.783659 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:17.783731 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 07:20:17.783975 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":153,"threads_started":1}
I20260926 07:20:17.785418 6912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7: Bootstrap starting.
I20260926 07:20:17.787847 6912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7: Neither blocks nor log segments found. Creating new log.
I20260926 07:20:17.788573 6912 log.cc:824] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:20:17.790910 6912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7: No bootstrap required, opened a new log
I20260926 07:20:17.794938 6912 raft_consensus.cc:348] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43513 } }
I20260926 07:20:17.795212 6912 raft_consensus.cc:374] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:20:17.795256 6912 raft_consensus.cc:729] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3dddc1789c42ff87bbf4fde4ad7ed7, State: Initialized, Role: FOLLOWER
I20260926 07:20:17.795744 6912 consensus_queue.cc:260] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43513 } }
I20260926 07:20:17.795881 6912 raft_consensus.cc:388] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:20:17.795928 6912 raft_consensus.cc:482] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:20:17.795998 6912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:20:17.796998 6912 raft_consensus.cc:504] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43513 } }
I20260926 07:20:17.797422 6912 leader_election.cc:302] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af3dddc1789c42ff87bbf4fde4ad7ed7; no voters:
I20260926 07:20:17.797825 6912 leader_election.cc:288] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:20:17.797962 6917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:20:17.798276 6917 raft_consensus.cc:686] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [term 1 LEADER]: Becoming Leader. State: Replica: af3dddc1789c42ff87bbf4fde4ad7ed7, State: Running, Role: LEADER
I20260926 07:20:17.798602 6917 consensus_queue.cc:237] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43513 } }
I20260926 07:20:17.799263 6912 sys_catalog.cc:564] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:20:17.800367 6919 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af3dddc1789c42ff87bbf4fde4ad7ed7. Latest consensus state: current_term: 1 leader_uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43513 } } }
I20260926 07:20:17.800371 6918 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3dddc1789c42ff87bbf4fde4ad7ed7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43513 } } }
I20260926 07:20:17.800531 6919 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [sys.catalog]: This master's current role is: LEADER
I20260926 07:20:17.800531 6918 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7 [sys.catalog]: This master's current role is: LEADER
I20260926 07:20:17.800962 6926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:20:17.804404 6926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:20:17.811036 6926 catalog_manager.cc:1347] Generated new cluster ID: 2f6dbfef77164b4b9f3fec7f70f5c83d
I20260926 07:20:17.811158 6926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:20:17.827111 6926 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:20:17.827517 6926 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:20:17.838703 6926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af3dddc1789c42ff87bbf4fde4ad7ed7: Generated new TSK 0
I20260926 07:20:17.839623 6926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:20:17.844801 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43513
--builtin_ntp_servers=127.28.235.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:20:17.970783 6936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:17.971119 6936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:17.971186 6936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:17.975634 6936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:17.975713 6936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:17.975773 6936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:20:17.981269 6936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:17.982550 6936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:17.984064 6936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:17.994663 6944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:17.994664 6942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:17.994664 6941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:17.994712 6936 server_base.cc:1034] running on GCE node
I20260926 07:20:17.995517 6936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:17.996223 6936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:17.997421 6936 hybrid_clock.cc:648] HybridClock initialized: now 1790407217997399 us; error 35 us; skew 500 ppm
I20260926 07:20:17.999769 6936 webserver.cc:460] Webserver started at http://127.28.235.193:42863/ using document root <none> and password file <none>
I20260926 07:20:18.000522 6936 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:18.000593 6936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:18.000773 6936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:18.003237 6936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "b3bded67344549fe81559ef93f210614"
format_stamp: "Formatted at 2026-09-26 07:20:18 on dist-test-slave-7zlw"
server_key: "20a51498d114b6e238d18898374c1b9a"
server_key_iv: "80a4ab7e95087fe7784f9e7ca500101b"
server_key_version: "encryptionkey@0"
I20260926 07:20:18.003842 6936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "b3bded67344549fe81559ef93f210614"
format_stamp: "Formatted at 2026-09-26 07:20:18 on dist-test-slave-7zlw"
server_key: "20a51498d114b6e238d18898374c1b9a"
server_key_iv: "80a4ab7e95087fe7784f9e7ca500101b"
server_key_version: "encryptionkey@0"
I20260926 07:20:18.008054 6936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:20:18.010934 6950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:18.012265 6936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:20:18.012374 6936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "b3bded67344549fe81559ef93f210614"
format_stamp: "Formatted at 2026-09-26 07:20:18 on dist-test-slave-7zlw"
server_key: "20a51498d114b6e238d18898374c1b9a"
server_key_iv: "80a4ab7e95087fe7784f9e7ca500101b"
server_key_version: "encryptionkey@0"
I20260926 07:20:18.012501 6936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:18.017118 6936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:18.017745 6936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:18.017946 6936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:18.018530 6936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:18.019500 6936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:18.019548 6936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:18.019588 6936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:18.019631 6936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:18.029156 6936 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:45477
I20260926 07:20:18.029184 7063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:45477 every 8 connection(s)
I20260926 07:20:18.030259 6936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:20:18.032666 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 6936
I20260926 07:20:18.032802 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:20:18.033179 29615 external_mini_cluster.cc:1428] Setting key 0a8f3eb2fb3e9cc812fba2b21d6631b0
I20260926 07:20:18.035172 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43513
--builtin_ntp_servers=127.28.235.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 07:20:18.037358 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:18.032462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:50681 (local address 127.28.235.254:43513)
0926 07:20:18.032687 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:18.032692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:18.033941 (+ 1249us) server_negotiation.cc:407] Connection header received
0926 07:20:18.035328 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:18.035332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:18.035373 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:18.035458 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:18.036775 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:18.036779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:18.036781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:20:18.036794 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:18.036820 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:20:18.036830 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:18.036999 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:18.037180 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 07:20:18.037227 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 07:20:18.038461 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:18.032958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43513 (local address 127.28.235.193:50681)
0926 07:20:18.033784 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:18.033839 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:18.035104 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:18.035629 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:18.035636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:18.035731 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:18.036400 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:20:18.036404 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:18.036472 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:20:18.036578 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:18.037001 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:18.037031 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:18.037037 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:20:18.037223 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 07:20:18.037522 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":137,"threads_started":1}
I20260926 07:20:18.039925 7064 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43513
I20260926 07:20:18.040257 7064 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:18.040710 7064 heartbeater.cc:511] Master 127.28.235.254:43513 requested a full tablet report, sending...
I20260926 07:20:18.042433 6876 ts_manager.cc:194] Registered new tserver with Master: b3bded67344549fe81559ef93f210614 (127.28.235.193:45477)
W20260926 07:20:18.156239 7067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:18.156599 7067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:18.156677 7067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:18.161566 7067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:18.161650 7067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:18.161746 7067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:20:18.167585 7067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:18.168751 7067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:18.170287 7067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:18.181277 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:18.181294 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:18.181334 7067 server_base.cc:1034] running on GCE node
W20260926 07:20:18.181293 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:18.181984 7067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:18.182723 7067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:18.184046 7067 hybrid_clock.cc:648] HybridClock initialized: now 1790407218184029 us; error 40 us; skew 500 ppm
I20260926 07:20:18.186144 7067 webserver.cc:460] Webserver started at http://127.28.235.194:42775/ using document root <none> and password file <none>
I20260926 07:20:18.186772 7067 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:18.186873 7067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:18.187122 7067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:18.189361 7067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "bd18cda7ea59466daff76edfb6d478b4"
format_stamp: "Formatted at 2026-09-26 07:20:18 on dist-test-slave-7zlw"
server_key: "4d311b9c333ddbcf88fe1da82b7fa74c"
server_key_iv: "d5db82550630ca4a2f8070179a44b6bf"
server_key_version: "encryptionkey@0"
I20260926 07:20:18.189976 7067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "bd18cda7ea59466daff76edfb6d478b4"
format_stamp: "Formatted at 2026-09-26 07:20:18 on dist-test-slave-7zlw"
server_key: "4d311b9c333ddbcf88fe1da82b7fa74c"
server_key_iv: "d5db82550630ca4a2f8070179a44b6bf"
server_key_version: "encryptionkey@0"
I20260926 07:20:18.194139 7067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:20:18.196949 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:18.198287 7067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:20:18.198433 7067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "bd18cda7ea59466daff76edfb6d478b4"
format_stamp: "Formatted at 2026-09-26 07:20:18 on dist-test-slave-7zlw"
server_key: "4d311b9c333ddbcf88fe1da82b7fa74c"
server_key_iv: "d5db82550630ca4a2f8070179a44b6bf"
server_key_version: "encryptionkey@0"
I20260926 07:20:18.198577 7067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:18.203397 7067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:18.204087 7067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:18.204286 7067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:18.204874 7067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:18.205891 7067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:18.205968 7067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:18.206036 7067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:18.206084 7067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:18.215853 7067 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:36687
I20260926 07:20:18.215871 7194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:36687 every 8 connection(s)
I20260926 07:20:18.216977 7067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:20:18.223060 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 7067
I20260926 07:20:18.223232 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:20:18.223574 29615 external_mini_cluster.cc:1428] Setting key 671b31b61917f1e5a2d4378201558d66
I20260926 07:20:18.224548 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:18.219419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:49395 (local address 127.28.235.254:43513)
0926 07:20:18.219606 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:18.219634 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:18.220956 (+ 1322us) server_negotiation.cc:407] Connection header received
0926 07:20:18.222383 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:18.222386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:18.222437 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:18.222522 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:18.223919 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:18.223923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:18.223926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:20:18.223944 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:18.223973 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:20:18.223985 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:18.224113 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:18.224353 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 07:20:18.224407 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:20:18.225718 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:18.219879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43513 (local address 127.28.235.194:49395)
0926 07:20:18.220787 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:18.220832 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:18.222151 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:18.222748 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:18.222756 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:18.222852 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:18.223569 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:20:18.223570 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:18.223615 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:20:18.223725 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:18.224099 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:18.224109 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:18.224114 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:20:18.224313 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 07:20:18.224572 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":128,"threads_started":1}
I20260926 07:20:18.226069 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43513
--builtin_ntp_servers=127.28.235.212:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 07:20:18.227218 7195 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43513
I20260926 07:20:18.227555 7195 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:18.228106 7195 heartbeater.cc:511] Master 127.28.235.254:43513 requested a full tablet report, sending...
I20260926 07:20:18.229590 6876 ts_manager.cc:194] Registered new tserver with Master: bd18cda7ea59466daff76edfb6d478b4 (127.28.235.194:36687)
W20260926 07:20:18.354947 7198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:18.355329 7198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:18.355422 7198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:18.359920 7198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:18.360037 7198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:18.360136 7198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:20:18.365748 7198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:18.367041 7198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:18.368661 7198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:18.379483 7204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:18.379493 7203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:18.379552 7206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:18.379757 7198 server_base.cc:1034] running on GCE node
I20260926 07:20:18.380321 7198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:18.381125 7198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:18.382349 7198 hybrid_clock.cc:648] HybridClock initialized: now 1790407218382325 us; error 47 us; skew 500 ppm
I20260926 07:20:18.384775 7198 webserver.cc:460] Webserver started at http://127.28.235.195:33613/ using document root <none> and password file <none>
I20260926 07:20:18.385576 7198 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:18.385689 7198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:18.385955 7198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:18.388350 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "1593fdfc775c4616808f88e8edc43e17"
format_stamp: "Formatted at 2026-09-26 07:20:18 on dist-test-slave-7zlw"
server_key: "1a68abf36455e2d13344664f7f6338a5"
server_key_iv: "daf69602c416e9b21ac2fa558fadd8ae"
server_key_version: "encryptionkey@0"
I20260926 07:20:18.389056 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "1593fdfc775c4616808f88e8edc43e17"
format_stamp: "Formatted at 2026-09-26 07:20:18 on dist-test-slave-7zlw"
server_key: "1a68abf36455e2d13344664f7f6338a5"
server_key_iv: "daf69602c416e9b21ac2fa558fadd8ae"
server_key_version: "encryptionkey@0"
I20260926 07:20:18.393493 7198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:20:18.396557 7212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:18.398154 7198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:20:18.398309 7198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "1593fdfc775c4616808f88e8edc43e17"
format_stamp: "Formatted at 2026-09-26 07:20:18 on dist-test-slave-7zlw"
server_key: "1a68abf36455e2d13344664f7f6338a5"
server_key_iv: "daf69602c416e9b21ac2fa558fadd8ae"
server_key_version: "encryptionkey@0"
I20260926 07:20:18.398468 7198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:18.404119 7198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:18.404843 7198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:18.405128 7198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:18.405736 7198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:18.406757 7198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:18.406836 7198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:18.406909 7198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:18.406947 7198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:18.416761 7198 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:41103
I20260926 07:20:18.416781 7325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:41103 every 8 connection(s)
I20260926 07:20:18.417850 7198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:20:18.424960 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:18.420021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:56645 (local address 127.28.235.254:43513)
0926 07:20:18.420173 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:18.420177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:18.421577 (+ 1400us) server_negotiation.cc:407] Connection header received
0926 07:20:18.422954 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:18.422958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:18.423004 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:18.423118 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:18.424409 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:18.424412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:18.424414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:20:18.424427 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:18.424450 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:20:18.424458 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:18.424551 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:18.424778 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 07:20:18.424827 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:20:18.425079 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 7198
I20260926 07:20:18.425184 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:20:18.425549 29615 external_mini_cluster.cc:1428] Setting key 304281d94e7fc8fb196e4c655549128f
I20260926 07:20:18.426241 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:18.420462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43513 (local address 127.28.235.195:56645)
0926 07:20:18.421380 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:18.421432 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:18.422711 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:18.423297 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:18.423305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:18.423403 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:18.424072 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:20:18.424073 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:18.424134 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:20:18.424211 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:18.424568 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:18.424577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:18.424582 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:20:18.424796 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 07:20:18.425123 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":151,"threads_started":1}
I20260926 07:20:18.428378 7326 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43513
I20260926 07:20:18.428694 7326 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:18.429345 7326 heartbeater.cc:511] Master 127.28.235.254:43513 requested a full tablet report, sending...
I20260926 07:20:18.430650 6876 ts_manager.cc:194] Registered new tserver with Master: 1593fdfc775c4616808f88e8edc43e17 (127.28.235.195:41103)
I20260926 07:20:18.441293 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:20:18.445014 6915 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:20:18.444028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.28.235.254:43513)
0926 07:20:18.444200 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:18.444203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:18.444351 (+ 148us) server_negotiation.cc:407] Connection header received
0926 07:20:18.444477 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:18.444480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:18.444517 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:18.444597 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:18.444861 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57506: BlockingRecv error: recv got EOF from 127.0.0.1:57506 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 07:20:18.445271 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 6936
I20260926 07:20:18.451826 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 7067
I20260926 07:20:18.458601 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 7198
I20260926 07:20:18.464318 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 6845
2026-09-26T07:20:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (894 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:20:18 dist-test-slave-7zlw krb5kdc[7336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:20:18 dist-test-slave-7zlw krb5kdc[7336](info): set up 2 sockets
Sep 26 07:20:18 dist-test-slave-7zlw krb5kdc[7336](info): commencing operation
krb5kdc: starting...
W20260926 07:20:19.542109 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:20:19 dist-test-slave-7zlw krb5kdc[7336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:20:19.673820 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43129
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:43129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:20:19.790068 7349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:19.790369 7349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:19.790411 7349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:19.794688 7349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:20:19.794770 7349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:19.794791 7349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:19.794811 7349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:20:19.794827 7349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:20:19.800633 7349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:43129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:19.801880 7349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:19.802937 7349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:19.811990 7355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:19.812036 7357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:19.812017 7354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:19.812391 7349 server_base.cc:1034] running on GCE node
I20260926 07:20:19.813160 7349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:19.814419 7349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:19.815615 7349 hybrid_clock.cc:648] HybridClock initialized: now 1790407219815605 us; error 31 us; skew 500 ppm
Sep 26 07:20:19 dist-test-slave-7zlw krb5kdc[7336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407219, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:19.824172 7349 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:20:19.825495 7349 webserver.cc:460] Webserver started at http://127.28.235.254:37695/ using document root <none> and password file <none>
I20260926 07:20:19.826186 7349 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:19.826293 7349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:19.826558 7349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:19.828805 7349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "3a11fc39d095475aa110b3d97f0c78d7"
format_stamp: "Formatted at 2026-09-26 07:20:19 on dist-test-slave-7zlw"
server_key: "a89536f6d0631563a9c427ef19eb06d7"
server_key_iv: "9d155ef9621c8072f905ed938e3020b4"
server_key_version: "encryptionkey@0"
I20260926 07:20:19.829521 7349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "3a11fc39d095475aa110b3d97f0c78d7"
format_stamp: "Formatted at 2026-09-26 07:20:19 on dist-test-slave-7zlw"
server_key: "a89536f6d0631563a9c427ef19eb06d7"
server_key_iv: "9d155ef9621c8072f905ed938e3020b4"
server_key_version: "encryptionkey@0"
I20260926 07:20:19.833897 7349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:20:19.836650 7364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:19.838297 7349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:20:19.838488 7349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "3a11fc39d095475aa110b3d97f0c78d7"
format_stamp: "Formatted at 2026-09-26 07:20:19 on dist-test-slave-7zlw"
server_key: "a89536f6d0631563a9c427ef19eb06d7"
server_key_iv: "9d155ef9621c8072f905ed938e3020b4"
server_key_version: "encryptionkey@0"
I20260926 07:20:19.838640 7349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:19.870293 7349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:19.874114 7349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:19.874353 7349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:19.882872 7349 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:43129
I20260926 07:20:19.882936 7416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:43129 every 8 connection(s)
I20260926 07:20:19.884055 7349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:20:19.887714 7417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:20:19.891532 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 7349
I20260926 07:20:19.891652 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:20:19.892020 29615 external_mini_cluster.cc:1428] Setting key 82bf1cdcfa493f4983ee0dc533c12cfd
I20260926 07:20:19.894518 7417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7: Bootstrap starting.
I20260926 07:20:19.896946 7417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7: Neither blocks nor log segments found. Creating new log.
I20260926 07:20:19.897758 7417 log.cc:824] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:20:19.899806 7417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7: No bootstrap required, opened a new log
Sep 26 07:20:19 dist-test-slave-7zlw krb5kdc[7336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:20:19.904186 7417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11fc39d095475aa110b3d97f0c78d7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43129 } }
I20260926 07:20:19.904436 7417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:20:19.904471 7417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a11fc39d095475aa110b3d97f0c78d7, State: Initialized, Role: FOLLOWER
I20260926 07:20:19.905246 7417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11fc39d095475aa110b3d97f0c78d7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43129 } }
I20260926 07:20:19.905395 7417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:20:19.905485 7417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:20:19.905586 7417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:20:19.906922 7417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11fc39d095475aa110b3d97f0c78d7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43129 } }
I20260926 07:20:19.907321 7417 leader_election.cc:302] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a11fc39d095475aa110b3d97f0c78d7; no voters:
I20260926 07:20:19.907677 7417 leader_election.cc:288] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:20:19.907802 7422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:20:19.908113 7422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [term 1 LEADER]: Becoming Leader. State: Replica: 3a11fc39d095475aa110b3d97f0c78d7, State: Running, Role: LEADER
I20260926 07:20:19.908340 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:19.893773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.28.235.254:43129)
0926 07:20:19.894310 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:19.894326 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:19.894367 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:20:19.895151 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:19.895173 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:19.895570 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:19.895987 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:19.896839 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:19.898052 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:19.898957 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:19.899460 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:19.899641 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:19.902537 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:19.902550 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:19.902573 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:20:19.902606 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:19.904578 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:19.905158 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:19.905164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:19.905171 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:19.905324 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:19.905742 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:19.905746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:19.905747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:19.906326 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:19.906555 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:19.906849 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 07:20:19.907219 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":144,"threads_started":1}
I20260926 07:20:19.908469 7422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11fc39d095475aa110b3d97f0c78d7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43129 } }
I20260926 07:20:19.909083 7417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:20:19.910449 7423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a11fc39d095475aa110b3d97f0c78d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11fc39d095475aa110b3d97f0c78d7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43129 } } }
I20260926 07:20:19.910590 7423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [sys.catalog]: This master's current role is: LEADER
I20260926 07:20:19.910974 7424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a11fc39d095475aa110b3d97f0c78d7. Latest consensus state: current_term: 1 leader_uuid: "3a11fc39d095475aa110b3d97f0c78d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a11fc39d095475aa110b3d97f0c78d7" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43129 } } }
I20260926 07:20:19.911077 7424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7 [sys.catalog]: This master's current role is: LEADER
I20260926 07:20:19.911561 7431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:20:19.915047 7431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:20:19.921882 7431 catalog_manager.cc:1347] Generated new cluster ID: c3d0c098e54446ffa158a548ce76e6c7
I20260926 07:20:19.922053 7431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:20:19.933084 7431 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:20:19.934288 7431 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:20:19.951383 7431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7: Generated new TSK 0
I20260926 07:20:19.952287 7431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:20:19.989276 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43129
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:20:20.107909 7443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:20.108222 7443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:20.108268 7443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:20.113188 7443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:20.113274 7443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:20.113377 7443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:20:20.119086 7443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:20.120337 7443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:20.121477 7443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:20.131942 7448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:20.132025 7451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:20.131942 7449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:20.132210 7443 server_base.cc:1034] running on GCE node
I20260926 07:20:20.132707 7443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:20.133534 7443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:20.134712 7443 hybrid_clock.cc:648] HybridClock initialized: now 1790407220134658 us; error 68 us; skew 500 ppm
Sep 26 07:20:20 dist-test-slave-7zlw krb5kdc[7336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407220, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:20.137676 7443 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:20:20.138620 7443 webserver.cc:460] Webserver started at http://127.28.235.193:35991/ using document root <none> and password file <none>
I20260926 07:20:20.139211 7443 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:20.139304 7443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:20.139567 7443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:20.141776 7443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "7c3cb374e0e046dd8265c927e030e753"
format_stamp: "Formatted at 2026-09-26 07:20:20 on dist-test-slave-7zlw"
server_key: "4d16746788766bf3a891407fd2f945e4"
server_key_iv: "2563053167388f95dddc5b0742538f11"
server_key_version: "encryptionkey@0"
I20260926 07:20:20.142401 7443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "7c3cb374e0e046dd8265c927e030e753"
format_stamp: "Formatted at 2026-09-26 07:20:20 on dist-test-slave-7zlw"
server_key: "4d16746788766bf3a891407fd2f945e4"
server_key_iv: "2563053167388f95dddc5b0742538f11"
server_key_version: "encryptionkey@0"
I20260926 07:20:20.146696 7443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:20:20.149372 7458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:20.150727 7443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:20:20.150880 7443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "7c3cb374e0e046dd8265c927e030e753"
format_stamp: "Formatted at 2026-09-26 07:20:20 on dist-test-slave-7zlw"
server_key: "4d16746788766bf3a891407fd2f945e4"
server_key_iv: "2563053167388f95dddc5b0742538f11"
server_key_version: "encryptionkey@0"
I20260926 07:20:20.151011 7443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:20.165683 7443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:20.169845 7443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:20.170086 7443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:20.170909 7443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:20.172031 7443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:20.172081 7443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:20.172153 7443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:20.172183 7443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:20.181634 7443 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:46821
I20260926 07:20:20.181689 7571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:46821 every 8 connection(s)
I20260926 07:20:20.182613 7443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:20:20.186641 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 7443
I20260926 07:20:20.186798 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:20:20.187139 29615 external_mini_cluster.cc:1428] Setting key 673c5e4da25c41d982bb6a55f8d36fce
Sep 26 07:20:20 dist-test-slave-7zlw krb5kdc[7336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407220, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:20:20.198863 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.184582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:45001 (local address 127.28.235.254:43129)
0926 07:20:20.184783 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:20.184788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:20.185695 (+ 907us) server_negotiation.cc:407] Connection header received
0926 07:20:20.187746 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:20.187750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:20.187814 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:20.187937 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:20.189523 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.190060 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.191478 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.191739 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.191869 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.194764 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:20.194768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:20.194770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:20:20.194797 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:20.196511 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:20.197152 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:20.197156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:20.197157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:20.197241 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:20.197618 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:20.197621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:20.197623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:20.197861 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:20.198004 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:20.198536 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 07:20:20.198681 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:20:20.199934 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.184961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43129 (local address 127.28.235.193:45001)
0926 07:20:20.185532 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:20.185579 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:20.187483 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:20.188127 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:20.188134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:20.188532 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:20.189340 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.189365 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.190199 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.190206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.191326 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.191337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.191920 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.191925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.192133 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.192931 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:20.192992 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:20.194563 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:20.196663 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:20.196670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:20.196688 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:20.196971 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:20.197346 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:20.197349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:20.197351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:20.197507 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:20.197997 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:20.198005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:20.198282 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 07:20:20.198487 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 07:20:20.198844 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":142,"threads_started":1}
I20260926 07:20:20.201470 7572 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43129
I20260926 07:20:20.201774 7572 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:20.202503 7572 heartbeater.cc:511] Master 127.28.235.254:43129 requested a full tablet report, sending...
I20260926 07:20:20.204463 7381 ts_manager.cc:194] Registered new tserver with Master: 7c3cb374e0e046dd8265c927e030e753 (127.28.235.193:46821)
I20260926 07:20:20.206327 7381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:45001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:20:20.221397 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43129
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:20:20.338661 7577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:20.338989 7577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:20.339082 7577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:20.344065 7577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:20.344141 7577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:20.344228 7577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:20:20.350113 7577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:20.351338 7577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:20.352497 7577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:20.363272 7583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:20.363272 7585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:20.363469 7577 server_base.cc:1034] running on GCE node
W20260926 07:20:20.363313 7582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:20.364099 7577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:20.364843 7577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:20.366052 7577 hybrid_clock.cc:648] HybridClock initialized: now 1790407220366024 us; error 40 us; skew 500 ppm
Sep 26 07:20:20 dist-test-slave-7zlw krb5kdc[7336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407220, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:20.368983 7577 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:20:20.370026 7577 webserver.cc:460] Webserver started at http://127.28.235.194:32823/ using document root <none> and password file <none>
I20260926 07:20:20.370637 7577 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:20.370741 7577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:20.370988 7577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:20.373250 7577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "6da74e4dc2144b35846864af620cf38f"
format_stamp: "Formatted at 2026-09-26 07:20:20 on dist-test-slave-7zlw"
server_key: "e87539e7f22649d40119f65f9f8522bb"
server_key_iv: "9e5f2e363f7e29e0b9bcdf04c39c46eb"
server_key_version: "encryptionkey@0"
I20260926 07:20:20.373888 7577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "6da74e4dc2144b35846864af620cf38f"
format_stamp: "Formatted at 2026-09-26 07:20:20 on dist-test-slave-7zlw"
server_key: "e87539e7f22649d40119f65f9f8522bb"
server_key_iv: "9e5f2e363f7e29e0b9bcdf04c39c46eb"
server_key_version: "encryptionkey@0"
I20260926 07:20:20.378018 7577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:20:20.380713 7592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:20.382228 7577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:20:20.382380 7577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "6da74e4dc2144b35846864af620cf38f"
format_stamp: "Formatted at 2026-09-26 07:20:20 on dist-test-slave-7zlw"
server_key: "e87539e7f22649d40119f65f9f8522bb"
server_key_iv: "9e5f2e363f7e29e0b9bcdf04c39c46eb"
server_key_version: "encryptionkey@0"
I20260926 07:20:20.382530 7577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:20.413149 7577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:20.416716 7577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:20.416941 7577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:20.417600 7577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:20.418454 7577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:20.418524 7577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:20.418612 7577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:20.418655 7577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:20.428056 7577 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:34149
I20260926 07:20:20.428081 7705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:34149 every 8 connection(s)
I20260926 07:20:20.429077 7577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:20:20.439159 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 7577
I20260926 07:20:20.439316 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:20:20.439630 29615 external_mini_cluster.cc:1428] Setting key c25f13cdd80c63fe2b33dc75b5af0891
Sep 26 07:20:20 dist-test-slave-7zlw krb5kdc[7336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407220, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:20:20.444762 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.431319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:45741 (local address 127.28.235.254:43129)
0926 07:20:20.431500 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:20.431504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:20.432464 (+ 960us) server_negotiation.cc:407] Connection header received
0926 07:20:20.434098 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:20.434102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:20.434164 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:20.434281 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:20.435544 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.436097 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.437311 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.437549 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.437665 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.440453 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:20.440457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:20.440460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:20:20.440499 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:20.442212 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:20.442957 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:20.442960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:20.442962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:20.443053 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:20.443435 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:20.443440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:20.443442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:20.443690 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:20.443834 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:20.444504 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 07:20:20.444633 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 07:20:20.446161 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.431738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43129 (local address 127.28.235.194:45741)
0926 07:20:20.432292 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:20.432340 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:20.433806 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:20.434474 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:20.434484 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:20.434866 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:20.435375 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.435387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.436230 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.436234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.437172 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.437182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.437722 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.437726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.437902 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.438536 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:20.438553 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:20.440265 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:20.442430 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:20.442437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:20.442463 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:20.442844 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:20.443159 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:20.443165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:20.443167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:20.443326 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:20.443894 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:20.443899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:20.444213 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 07:20:20.444447 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 07:20:20.444850 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":169,"threads_started":1}
I20260926 07:20:20.447603 7706 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43129
I20260926 07:20:20.447929 7706 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:20.448601 7706 heartbeater.cc:511] Master 127.28.235.254:43129 requested a full tablet report, sending...
I20260926 07:20:20.450012 7381 ts_manager.cc:194] Registered new tserver with Master: 6da74e4dc2144b35846864af620cf38f (127.28.235.194:34149)
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:20:20.450707 7381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:45741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:20:20.478434 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43129
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:20:20.594594 7711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:20:20.594867 7711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:20:20.594911 7711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:20:20.598866 7711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:20:20.598924 7711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:20:20.598982 7711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:20:20.604290 7711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:20:20.605371 7711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:20:20.606357 7711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:20:20.616677 7717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:20.616742 7711 server_base.cc:1034] running on GCE node
W20260926 07:20:20.616685 7716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:20:20.616685 7719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:20:20.617393 7711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:20:20.618052 7711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:20:20.619251 7711 hybrid_clock.cc:648] HybridClock initialized: now 1790407220619245 us; error 45 us; skew 500 ppm
Sep 26 07:20:20 dist-test-slave-7zlw krb5kdc[7336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407220, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:20:20.622061 7711 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:20:20.622959 7711 webserver.cc:460] Webserver started at http://127.28.235.195:33767/ using document root <none> and password file <none>
I20260926 07:20:20.623497 7711 fs_manager.cc:359] Metadata directory not provided
I20260926 07:20:20.623553 7711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:20:20.623718 7711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:20:20.625870 7711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "95b4f1931f93493789193a451a2e6507"
format_stamp: "Formatted at 2026-09-26 07:20:20 on dist-test-slave-7zlw"
server_key: "55caebfae40ce632037cc51e8f4b1e63"
server_key_iv: "710028b64b3a98b0ce7a19ef62df45dd"
server_key_version: "encryptionkey@0"
I20260926 07:20:20.626554 7711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "95b4f1931f93493789193a451a2e6507"
format_stamp: "Formatted at 2026-09-26 07:20:20 on dist-test-slave-7zlw"
server_key: "55caebfae40ce632037cc51e8f4b1e63"
server_key_iv: "710028b64b3a98b0ce7a19ef62df45dd"
server_key_version: "encryptionkey@0"
I20260926 07:20:20.630509 7711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:20:20.633167 7726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:20.634399 7711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:20:20.634536 7711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "95b4f1931f93493789193a451a2e6507"
format_stamp: "Formatted at 2026-09-26 07:20:20 on dist-test-slave-7zlw"
server_key: "55caebfae40ce632037cc51e8f4b1e63"
server_key_iv: "710028b64b3a98b0ce7a19ef62df45dd"
server_key_version: "encryptionkey@0"
I20260926 07:20:20.634652 7711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:20:20.649784 7711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:20:20.653214 7711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:20:20.653467 7711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:20:20.654156 7711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:20:20.655228 7711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:20:20.655287 7711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:20.655392 7711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:20:20.655452 7711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:20:20.666574 7711 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:42121
I20260926 07:20:20.666594 7839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:42121 every 8 connection(s)
I20260926 07:20:20.667618 7711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:20:20.675431 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 7711
I20260926 07:20:20.675573 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:20:20.675885 29615 external_mini_cluster.cc:1428] Setting key 7fe0c1d0ce26cc182956ef34a5613449
Sep 26 07:20:20 dist-test-slave-7zlw krb5kdc[7336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407220, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:20:20.682992 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.669655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:54877 (local address 127.28.235.254:43129)
0926 07:20:20.669817 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:20.669822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:20.670738 (+ 916us) server_negotiation.cc:407] Connection header received
0926 07:20:20.672122 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:20.672129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:20.672181 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:20.672263 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:20.673686 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.674605 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.676083 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.676330 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.676446 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.679338 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:20.679341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:20.679343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:20:20.679369 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:20.680797 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:20.681405 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:20.681408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:20.681409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:20.681489 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:20.681839 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:20.681842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:20.681843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:20.682116 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:20.682208 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:20.682712 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 07:20:20.682815 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:20:20.683784 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.670106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43129 (local address 127.28.235.195:54877)
0926 07:20:20.670585 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:20.670626 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:20.671894 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:20.672520 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:20.672527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:20.672973 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:20:20.673539 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.673554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.674765 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.674768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.675955 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.675984 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.676486 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.676491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.676671 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.677380 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:20:20.677401 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:20:20.679162 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:20:20.680918 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:20.680923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:20.680967 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:20.681312 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:20.681571 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:20:20.681573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:20:20.681575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:20:20.681706 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:20:20.682221 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:20:20.682227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:20:20.682474 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 07:20:20.682626 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 07:20:20.682884 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
I20260926 07:20:20.685075 7840 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43129
I20260926 07:20:20.685338 7840 heartbeater.cc:464] Registering TS with master...
I20260926 07:20:20.685896 7840 heartbeater.cc:511] Master 127.28.235.254:43129 requested a full tablet report, sending...
I20260926 07:20:20.686967 7381 ts_manager.cc:194] Registered new tserver with Master: 95b4f1931f93493789193a451a2e6507 (127.28.235.195:42121)
I20260926 07:20:20.687572 7381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:54877
I20260926 07:20:20.689867 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:20:20.698860 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.691675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.28.235.254:43129)
0926 07:20:20.691790 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:20.691794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:20.691923 (+ 129us) server_negotiation.cc:407] Connection header received
0926 07:20:20.692030 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:20.692032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:20.692069 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:20.692129 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:20:20.692754 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.693340 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.694226 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.694429 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.694517 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.695897 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:20:20.695900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:20:20.695902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:20:20.695930 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:20:20.697296 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:20.697745 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:20.697748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:20.697749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:20.697824 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:20:20.698096 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:20:20.698098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:20:20.698099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:20:20.698306 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:20:20.698409 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:20.698644 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 07:20:20.698742 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:20:20.702032 7381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:20:20.705179 7381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:20:20.720960 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.715881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.28.235.195:42121)
0926 07:20:20.716287 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:20.716300 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:20.716321 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:20:20.716388 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:20.716392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:20.716585 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:20.716738 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:20:20.716973 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.718087 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.719615 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.720160 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.720298 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.720446 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:20.720657 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 07:20:20.720767 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":147,"threads_started":1}
I20260926 07:20:20.721179 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.714931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.28.235.193:46821)
0926 07:20:20.715597 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:20.715604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:20.715627 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:20:20.715816 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:20.715823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:20.716045 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:20.716180 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:20:20.716649 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.718073 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.719365 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.720200 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.720323 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.720483 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:20.720914 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 07:20:20.721039 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":83,"threads_started":1}
I20260926 07:20:20.721307 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.714660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.28.235.194:34149)
0926 07:20:20.715079 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:20.715090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:20.715481 (+ 391us) server_negotiation.cc:407] Connection header received
0926 07:20:20.715787 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:20.715795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:20.716030 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:20.716171 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:20:20.716619 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.717593 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.719459 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.720206 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.720323 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.720445 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:20.721051 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 07:20:20.721149 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":109,"threads_started":1}
I20260926 07:20:20.720949 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.715610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:42121 (local address 127.0.0.1:47246)
0926 07:20:20.715952 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:20.715965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:20.716041 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:20.716733 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:20.716736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:20.716755 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:20:20.716847 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.716853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.718221 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.718223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.719506 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.719514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.720297 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.720301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.720415 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.720466 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:20:20.720625 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 07:20:20.720774 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":89,"thread_start_us":96,"threads_started":1}
I20260926 07:20:20.721827 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.714527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:34149 (local address 127.0.0.1:39542)
0926 07:20:20.715314 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:20.715367 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:20.715598 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:20.716351 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:20.716355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:20.716383 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:20:20.716492 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.716497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.717750 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.717753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.719333 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.719344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.720528 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.720532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.720720 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.720763 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:20:20.720835 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 07:20:20.720926 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":183,"spinlock_wait_cycles":2432,"thread_start_us":95,"threads_started":1}
I20260926 07:20:20.721567 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:20.714610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:46821 (local address 127.0.0.1:43742)
0926 07:20:20.715315 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:20.715367 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:20.715610 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:20.716352 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:20.716357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:20.716391 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:20:20.716510 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.716517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.718205 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.718211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.719230 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:20.719240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:20.720529 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:20.720532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:20.720727 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:20.720764 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:20:20.720835 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:20:20.721404 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":587,"spinlock_wait_cycles":7296,"thread_start_us":124,"threads_started":1}
I20260926 07:20:20.724466 7774 tablet_service.cc:1467] Processing CreateTablet for tablet af2a3e2e9fb54bac8994d1b9546c7be7 (DEFAULT_TABLE table=test-table [id=6220db532f2441269e4a6c7158b179d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:20:20.724466 7506 tablet_service.cc:1467] Processing CreateTablet for tablet af2a3e2e9fb54bac8994d1b9546c7be7 (DEFAULT_TABLE table=test-table [id=6220db532f2441269e4a6c7158b179d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:20:20.725603 7506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2a3e2e9fb54bac8994d1b9546c7be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:20:20.725612 7774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2a3e2e9fb54bac8994d1b9546c7be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:20:20.727543 7640 tablet_service.cc:1467] Processing CreateTablet for tablet af2a3e2e9fb54bac8994d1b9546c7be7 (DEFAULT_TABLE table=test-table [id=6220db532f2441269e4a6c7158b179d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:20:20.728711 7640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2a3e2e9fb54bac8994d1b9546c7be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:20:20.732517 7857 tablet_bootstrap.cc:492] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753: Bootstrap starting.
I20260926 07:20:20.733763 7858 tablet_bootstrap.cc:492] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: Bootstrap starting.
I20260926 07:20:20.734820 7857 tablet_bootstrap.cc:654] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753: Neither blocks nor log segments found. Creating new log.
I20260926 07:20:20.735579 7857 log.cc:824] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753: Log is configured to *not* fsync() on all Append() calls
I20260926 07:20:20.736372 7858 tablet_bootstrap.cc:654] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: Neither blocks nor log segments found. Creating new log.
I20260926 07:20:20.737267 7858 log.cc:824] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: Log is configured to *not* fsync() on all Append() calls
I20260926 07:20:20.737493 7860 tablet_bootstrap.cc:492] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f: Bootstrap starting.
I20260926 07:20:20.738059 7857 tablet_bootstrap.cc:492] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753: No bootstrap required, opened a new log
I20260926 07:20:20.738266 7857 ts_tablet_manager.cc:1397] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:20:20.739195 7858 tablet_bootstrap.cc:492] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: No bootstrap required, opened a new log
I20260926 07:20:20.739451 7858 ts_tablet_manager.cc:1397] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:20:20.740351 7860 tablet_bootstrap.cc:654] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f: Neither blocks nor log segments found. Creating new log.
I20260926 07:20:20.741274 7860 log.cc:824] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:20:20.742879 7858 raft_consensus.cc:348] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 } } peers { permanent_uuid: "7c3cb374e0e046dd8265c927e030e753" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46821 } } peers { permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 } }
I20260926 07:20:20.742916 7857 raft_consensus.cc:348] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 } } peers { permanent_uuid: "7c3cb374e0e046dd8265c927e030e753" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46821 } } peers { permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 } }
I20260926 07:20:20.743191 7858 raft_consensus.cc:374] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:20:20.743191 7857 raft_consensus.cc:374] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:20:20.743248 7857 raft_consensus.cc:729] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3cb374e0e046dd8265c927e030e753, State: Initialized, Role: FOLLOWER
I20260926 07:20:20.743256 7858 raft_consensus.cc:729] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b4f1931f93493789193a451a2e6507, State: Initialized, Role: FOLLOWER
I20260926 07:20:20.743734 7858 consensus_queue.cc:260] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 } } peers { permanent_uuid: "7c3cb374e0e046dd8265c927e030e753" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46821 } } peers { permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 } }
I20260926 07:20:20.744001 7857 consensus_queue.cc:260] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 } } peers { permanent_uuid: "7c3cb374e0e046dd8265c927e030e753" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46821 } } peers { permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 } }
I20260926 07:20:20.744316 7860 tablet_bootstrap.cc:492] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f: No bootstrap required, opened a new log
I20260926 07:20:20.744536 7860 ts_tablet_manager.cc:1397] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260926 07:20:20.744807 7572 heartbeater.cc:503] Master 127.28.235.254:43129 was elected leader, sending a full tablet report...
I20260926 07:20:20.744836 7858 ts_tablet_manager.cc:1428] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 07:20:20.744988 7857 ts_tablet_manager.cc:1428] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:20:20.745175 7840 heartbeater.cc:503] Master 127.28.235.254:43129 was elected leader, sending a full tablet report...
I20260926 07:20:20.747843 7860 raft_consensus.cc:348] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 } } peers { permanent_uuid: "7c3cb374e0e046dd8265c927e030e753" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46821 } } peers { permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 } }
I20260926 07:20:20.748122 7860 raft_consensus.cc:374] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:20:20.748178 7860 raft_consensus.cc:729] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da74e4dc2144b35846864af620cf38f, State: Initialized, Role: FOLLOWER
I20260926 07:20:20.748672 7860 consensus_queue.cc:260] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 } } peers { permanent_uuid: "7c3cb374e0e046dd8265c927e030e753" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46821 } } peers { permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 } }
I20260926 07:20:20.749421 7706 heartbeater.cc:503] Master 127.28.235.254:43129 was elected leader, sending a full tablet report...
I20260926 07:20:20.749835 7860 ts_tablet_manager.cc:1428] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260926 07:20:20.918799 7841 tablet.cc:2369] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:20:20.930393 7707 tablet.cc:2369] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:20:20.933957 7573 tablet.cc:2369] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:20:21.054167 7864 raft_consensus.cc:482] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:20:21.054526 7864 raft_consensus.cc:504] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 } } peers { permanent_uuid: "7c3cb374e0e046dd8265c927e030e753" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46821 } } peers { permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 } }
I20260926 07:20:21.055815 7864 leader_election.cc:288] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6da74e4dc2144b35846864af620cf38f (127.28.235.194:34149), 95b4f1931f93493789193a451a2e6507 (127.28.235.195:42121)
I20260926 07:20:21.059885 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:21.056305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:55297 (local address 127.28.235.195:42121)
0926 07:20:21.056436 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:21.056442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:21.056638 (+ 196us) server_negotiation.cc:407] Connection header received
0926 07:20:21.056813 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:21.056817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:21.056874 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:21.056957 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:20:21.057275 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:21.057784 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:21.058889 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:21.059291 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:21.059432 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:21.059488 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:21.059674 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 07:20:21.059746 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:20:21.059885 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:21.056194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:50099 (local address 127.28.235.194:34149)
0926 07:20:21.056333 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:21.056338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:21.056719 (+ 381us) server_negotiation.cc:407] Connection header received
0926 07:20:21.056872 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:21.056875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:21.056929 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:21.057105 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:20:21.057370 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:21.057861 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:21.058913 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:21.059303 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:21.059432 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:21.059489 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:21.059685 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 07:20:21.059759 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:20:21.059914 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:21.056115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:34149 (local address 127.28.235.193:50099)
0926 07:20:21.056609 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:21.056621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:21.056750 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:21.057114 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:21.057117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:21.057133 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:20:21.057253 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:21.057259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:21.057984 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:21.057991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:21.058784 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:21.058793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:21.059428 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:21.059432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:21.059562 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:21.059580 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:20:21.059669 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:20:21.059750 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":64,"thread_start_us":100,"threads_started":1}
I20260926 07:20:21.059927 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:21.056115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:42121 (local address 127.28.235.193:55297)
0926 07:20:21.056529 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:21.056542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:20:21.056689 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:20:21.056983 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:20:21.056986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:20:21.057001 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:20:21.057164 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:21.057169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:21.057905 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:21.057910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:21.058758 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:20:21.058767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:20:21.059426 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:20:21.059430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:20:21.059531 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:21.059561 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:20:21.059659 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:20:21.059741 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":3,"thread_start_us":134,"threads_started":1}
I20260926 07:20:21.060679 7660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2a3e2e9fb54bac8994d1b9546c7be7" candidate_uuid: "7c3cb374e0e046dd8265c927e030e753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da74e4dc2144b35846864af620cf38f" is_pre_election: true
I20260926 07:20:21.060762 7794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2a3e2e9fb54bac8994d1b9546c7be7" candidate_uuid: "7c3cb374e0e046dd8265c927e030e753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b4f1931f93493789193a451a2e6507" is_pre_election: true
I20260926 07:20:21.061055 7660 raft_consensus.cc:2454] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3cb374e0e046dd8265c927e030e753 in term 0.
I20260926 07:20:21.061190 7794 raft_consensus.cc:2454] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3cb374e0e046dd8265c927e030e753 in term 0.
I20260926 07:20:21.061784 7462 leader_election.cc:302] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6da74e4dc2144b35846864af620cf38f, 7c3cb374e0e046dd8265c927e030e753, 95b4f1931f93493789193a451a2e6507; no voters:
I20260926 07:20:21.062179 7864 raft_consensus.cc:2781] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:20:21.062314 7864 raft_consensus.cc:482] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:20:21.062405 7864 raft_consensus.cc:3037] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:20:21.063591 7864 raft_consensus.cc:504] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 } } peers { permanent_uuid: "7c3cb374e0e046dd8265c927e030e753" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46821 } } peers { permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 } }
I20260926 07:20:21.064044 7864 leader_election.cc:288] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [CANDIDATE]: Term 1 election: Requested vote from peers 6da74e4dc2144b35846864af620cf38f (127.28.235.194:34149), 95b4f1931f93493789193a451a2e6507 (127.28.235.195:42121)
I20260926 07:20:21.064580 7794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2a3e2e9fb54bac8994d1b9546c7be7" candidate_uuid: "7c3cb374e0e046dd8265c927e030e753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b4f1931f93493789193a451a2e6507"
I20260926 07:20:21.064599 7660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2a3e2e9fb54bac8994d1b9546c7be7" candidate_uuid: "7c3cb374e0e046dd8265c927e030e753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da74e4dc2144b35846864af620cf38f"
I20260926 07:20:21.064730 7660 raft_consensus.cc:3037] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:20:21.064730 7794 raft_consensus.cc:3037] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:20:21.066143 7660 raft_consensus.cc:2454] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3cb374e0e046dd8265c927e030e753 in term 1.
I20260926 07:20:21.066143 7794 raft_consensus.cc:2454] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3cb374e0e046dd8265c927e030e753 in term 1.
I20260926 07:20:21.066540 7461 leader_election.cc:302] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6da74e4dc2144b35846864af620cf38f, 7c3cb374e0e046dd8265c927e030e753; no voters:
I20260926 07:20:21.066741 7864 raft_consensus.cc:2781] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:20:21.067000 7864 raft_consensus.cc:686] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 1 LEADER]: Becoming Leader. State: Replica: 7c3cb374e0e046dd8265c927e030e753, State: Running, Role: LEADER
I20260926 07:20:21.067433 7864 consensus_queue.cc:237] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 } } peers { permanent_uuid: "7c3cb374e0e046dd8265c927e030e753" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46821 } } peers { permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 } }
I20260926 07:20:21.071130 7378 catalog_manager.cc:5495] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c3cb374e0e046dd8265c927e030e753 (127.28.235.193). New cstate: current_term: 1 leader_uuid: "7c3cb374e0e046dd8265c927e030e753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3cb374e0e046dd8265c927e030e753" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260926 07:20:21.084498 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:21.080910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.28.235.193:46821)
0926 07:20:21.081226 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 07:20:21.081230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:20:21.081243 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:20:21.081314 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:20:21.081316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:20:21.081371 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:20:21.081487 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:20:21.081702 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:21.082191 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:21.083122 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:20:21.083342 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:20:21.083443 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:20:21.083715 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:20:21.084161 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:20:21.084256 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:20:21.084312 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 07:20:21.084365 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 07:20:21.091257 7794 raft_consensus.cc:1261] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507 [term 1 FOLLOWER]: Refusing update from remote peer 7c3cb374e0e046dd8265c927e030e753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:20:21.091266 7660 raft_consensus.cc:1261] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f [term 1 FOLLOWER]: Refusing update from remote peer 7c3cb374e0e046dd8265c927e030e753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:20:21.092068 7868 consensus_queue.cc:1035] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da74e4dc2144b35846864af620cf38f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 34149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:20:21.092247 7864 consensus_queue.cc:1035] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b4f1931f93493789193a451a2e6507" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:20:21.102339 7872 mvcc.cc:205] Tried to move back new op lower bound from 7333507977580699648 to 7333507977494831104. Current Snapshot: MvccSnapshot[applied={T|T < 7333507977580699648}]
I20260926 07:20:21.112505 7378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:20:21.112680 7378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:20:21.115386 7378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a11fc39d095475aa110b3d97f0c78d7: Sending DeleteTablet for 3 replicas of tablet af2a3e2e9fb54bac8994d1b9546c7be7
I20260926 07:20:21.116346 7506 tablet_service.cc:1514] Processing DeleteTablet for tablet af2a3e2e9fb54bac8994d1b9546c7be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:20:21 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:43742
I20260926 07:20:21.116325 7640 tablet_service.cc:1514] Processing DeleteTablet for tablet af2a3e2e9fb54bac8994d1b9546c7be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:20:21 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:39542
I20260926 07:20:21.116895 7881 tablet_replica.cc:330] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753: stopping tablet replica
I20260926 07:20:21.116892 7774 tablet_service.cc:1514] Processing DeleteTablet for tablet af2a3e2e9fb54bac8994d1b9546c7be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:20:21 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:47246
I20260926 07:20:21.117276 7881 raft_consensus.cc:2229] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:20:21.117501 7883 tablet_replica.cc:330] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: stopping tablet replica
I20260926 07:20:21.117725 7881 raft_consensus.cc:2258] T af2a3e2e9fb54bac8994d1b9546c7be7 P 7c3cb374e0e046dd8265c927e030e753 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:20:21.117921 7883 raft_consensus.cc:2229] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:20:21.118355 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 7443
I20260926 07:20:21.119341 7883 raft_consensus.cc:2258] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:20:21.120875 7883 ts_tablet_manager.cc:1905] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:20:21.123179 7883 ts_tablet_manager.cc:1918] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:20:21.123266 7883 log.cc:1196] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/af2a3e2e9fb54bac8994d1b9546c7be7
I20260926 07:20:21.123539 7882 tablet_replica.cc:330] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f: stopping tablet replica
I20260926 07:20:21.123576 7883 ts_tablet_manager.cc:1939] T af2a3e2e9fb54bac8994d1b9546c7be7 P 95b4f1931f93493789193a451a2e6507: Deleting consensus metadata
I20260926 07:20:21.123916 7882 raft_consensus.cc:2229] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:20:21.124300 7882 raft_consensus.cc:2258] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:20:21.124959 7368 catalog_manager.cc:4841] TS 95b4f1931f93493789193a451a2e6507 (127.28.235.195:42121): tablet af2a3e2e9fb54bac8994d1b9546c7be7 (table test-table [id=6220db532f2441269e4a6c7158b179d2]) successfully deleted
W20260926 07:20:21.125638 7365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:46821 (error 108)
I20260926 07:20:21.125854 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 7577
I20260926 07:20:21.126022 7882 ts_tablet_manager.cc:1905] T af2a3e2e9fb54bac8994d1b9546c7be7 P 6da74e4dc2144b35846864af620cf38f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:20:21.126900 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:21.126539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:46821 (local address 127.0.0.1:43750)
0926 07:20:21.126708 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:21.126792 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:46821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 07:20:21.127126 7365 catalog_manager.cc:4584] TS 7c3cb374e0e046dd8265c927e030e753 (127.28.235.193:46821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af2a3e2e9fb54bac8994d1b9546c7be7: Network error: Client connection negotiation failed: client connection to 127.28.235.193:46821: connect: Connection refused (error 111)
I20260926 07:20:21.133170 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 7711
I20260926 07:20:21.134682 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:20:21.133404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:34149 (local address 127.0.0.1:39558)
0926 07:20:21.134500 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0926 07:20:21.134570 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.194:34149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":988}
I20260926 07:20:21.138638 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 7349
2026-09-26T07:20:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2675 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:20:21 dist-test-slave-7zlw krb5kdc[7887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:20:21 dist-test-slave-7zlw krb5kdc[7887](info): set up 2 sockets
Sep 26 07:20:21 dist-test-slave-7zlw krb5kdc[7887](info): commencing operation
krb5kdc: starting...
W20260926 07:20:22.214136 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:20:22 dist-test-slave-7zlw krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:20:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/rangeradmin_127.28.235.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/rangeradmin_127.28.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/rangerlookup_127.28.235.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/rangerlookup_127.28.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/HTTP_127.28.235.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/HTTP_127.28.235.212@KRBTEST.COM.keytab.
I20260926 07:20:22.401782 29615 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/postgres -l logfile start
2026-09-26 07:20:25.309 UTC [7917] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:20:25.309 UTC [7917] LOG: listening on IPv4 address "127.28.235.212", port 52729
2026-09-26 07:20:25.314 UTC [7917] LOG: listening on Unix socket "/tmp/.s.PGSQL.52729"
2026-09-26 07:20:25.350 UTC [7920] LOG: database system was shut down at 2026-09-26 07:20:23 UTC
2026-09-26 07:20:25.355 UTC [7917] LOG: database system is ready to accept connections
I20260926 07:20:26.318096 29615 mini_postgres.cc:96] Postgres bound to 52729
2026-09-26 07:20:26.324 UTC [7928] FATAL: database "slave" does not exist
127.28.235.212:52729 - accepting connections
I20260926 07:20:26.325420 29615 mini_ranger.cc:161] Starting Ranger...
I20260926 07:20:26.341701 29615 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:20:26.948589 29615 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:20:26.948693 29615 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-admin
2026-09-26 07:20:27,266 [I] DB FLAVOR :POSTGRES
2026-09-26 07:20:27,267 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:20:27,267 [I] Checking connection..
2026-09-26 07:20:27,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:20:27,509 [I] Checking connection passed.
2026-09-26 07:20:27,509 [I] --------- Verifying version history table ---------
2026-09-26 07:20:27,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:20:27,754 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:20:27,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:20:28,007 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:20:28,008 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:20:28,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:20:28.274 UTC [7992] WARNING: there is no transaction in progress
2026-09-26 07:20:28,281 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:20:28,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:20:28,535 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:20:28,535 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:20:28,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:20:28,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:20:29,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7zlw.c.gcp-upstream.internal','N') ;"
2026-09-26 07:20:29,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:20:29,517 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:20:29,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:20:29,776 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:20:29,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:20:30,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:20:30,289 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:20:30,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:20:31.463 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.483 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.504 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.523 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.543 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.565 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.632 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.641 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.661 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.675 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.688 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.706 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.718 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.728 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.738 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:31.748 UTC [8100] WARNING: there is no transaction in progress
2026-09-26 07:20:32,584 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:20:32,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:20:32,817 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:20:32,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:20:33,092 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:20:33,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:20:33,344 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:20:33,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:20:33,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:20:33,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7zlw.c.gcp-upstream.internal';"
2026-09-26 07:20:34,091 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:20:34,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:20:34,361 [I] DB_PATCHES have already been applied
I20260926 07:20:34.367504 29615 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:20:34.367574 29615 mini_ranger.cc:193] Using host: 127.28.235.212
I20260926 07:20:34.371181 29615 mini_ranger.cc:227] Ranger admin URL: http://127.28.235.212:43425
Sep 26, 2026 7:20:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:20:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:20:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:20:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:20:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:20:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:20:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:20:35.493810 29615 mini_ranger.cc:160] Time spent starting Ranger: real 9.168s user 0.004s sys 0.007s
Sep 26, 2026 7:20:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:20:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 7:20:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:20:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:20:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:20:58.040828 29615 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/rangerkms_127.28.235.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/rangerkms_127.28.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.235.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.235.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.235.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/HTTP_127.28.235.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.235.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/HTTP_127.28.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:20:58 dist-test-slave-7zlw krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407258, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:20:58.134393 29615 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:20:58.144939 29615 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:20:59.252966 29615 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:20:59.253144 29615 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-kms
I20260926 07:20:59.253280 29615 mini_ranger_kms.cc:227] Using postgres at 127.28.235.212:52729
2026-09-26 07:20:59,450 [I] DB FLAVOR :POSTGRES
2026-09-26 07:20:59,450 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:20:59,451 [I] Checking connection
2026-09-26 07:20:59,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:20:59,715 [I] connection success
2026-09-26 07:20:59,715 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:20:59,715 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:20:59,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:20:59,974 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:20:59,975 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:20:59,975 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:20:59,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:52729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:21:00,255 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:21:00.787909 29615 mini_ranger_kms.cc:310] Created kms service
I20260926 07:21:00.993039 29615 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:21:01.048264 29615 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:21:01 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407258, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:21:01.513116 29615 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:21:01.513458 29615 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:21:01.513482 29615 mini_ranger_kms.cc:240] Using host: 127.28.235.212
I20260926 07:21:01.518042 29615 mini_ranger_kms.cc:276] Ranger KMS PID: 8281
I20260926 07:21:01.518224 29615 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.235.212:52903
Sep 26, 2026 7:21:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:21:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:21:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:21:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:21:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:21:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 07:21:03.825625 29615 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.691s user 0.018s sys 0.162s
I20260926 07:21:03.825776 29615 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:21:03.825907 29615 mini_ranger_kms.cc:401] 127.28.235.212:52903/kms/v1/keys
Sep 26 07:21:03 dist-test-slave-7zlw krb5kdc[7887](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407263, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.235.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:21:04 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407263, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.235.212@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
Sep 26 07:21:04 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407263, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.235.212@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
Sep 26 07:21:07 dist-test-slave-7zlw krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:21:07.129804 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:39199
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:39199
--ranger_config_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:52903/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:21:07.264366 8331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:21:07.264739 8331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:21:07.264837 8331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:21:07.269555 8331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:21:07.269663 8331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:21:07.269701 8331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:21:07.269759 8331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:21:07.269879 8331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:21:07.276175 8331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.235.212:52903/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:39199
--ranger_config_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:39199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:21:07.277777 8331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:21:07.279210 8331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:21:07.290606 8337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:07.290679 8336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:07.291062 8339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:21:07.291038 8331 server_base.cc:1034] running on GCE node
I20260926 07:21:07.291688 8331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:21:07.293361 8331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:21:07.294595 8331 hybrid_clock.cc:648] HybridClock initialized: now 1790407267294548 us; error 39 us; skew 500 ppm
Sep 26 07:21:07 dist-test-slave-7zlw krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407267, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:21:07.297715 8331 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:21:07.298882 8331 webserver.cc:460] Webserver started at http://127.28.235.254:35729/ using document root <none> and password file <none>
I20260926 07:21:07.299490 8331 fs_manager.cc:359] Metadata directory not provided
I20260926 07:21:07.299575 8331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:21:07.299817 8331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:21:07 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407267, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:21:07.368664 8331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "fc7d657eed62472d9d36f8bef3adccfa"
format_stamp: "Formatted at 2026-09-26 07:21:07 on dist-test-slave-7zlw"
server_key: "153b9158cfb052e49770018a29d00d05"
server_key_iv: "bf1649b4501675040e2ff9701b9a073b"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:07.369382 8331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "fc7d657eed62472d9d36f8bef3adccfa"
format_stamp: "Formatted at 2026-09-26 07:21:07 on dist-test-slave-7zlw"
server_key: "153b9158cfb052e49770018a29d00d05"
server_key_iv: "bf1649b4501675040e2ff9701b9a073b"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:07.373677 8331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:21:07.389483 8346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:07.391095 8331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:21:07.391258 8331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "fc7d657eed62472d9d36f8bef3adccfa"
format_stamp: "Formatted at 2026-09-26 07:21:07 on dist-test-slave-7zlw"
server_key: "153b9158cfb052e49770018a29d00d05"
server_key_iv: "bf1649b4501675040e2ff9701b9a073b"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:07.391412 8331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:21:07.407279 8331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:21:07.410842 8331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:21:07.411064 8331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:21:07.420044 8331 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:39199
I20260926 07:21:07.420053 8398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:39199 every 8 connection(s)
I20260926 07:21:07.421190 8331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:21:07.425064 8399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:21:07.429679 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 8331
I20260926 07:21:07.429932 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:21:07.430255 29615 external_mini_cluster.cc:1428] Setting key 3f11bb72e59a78cebd5a2ba003fa272f
I20260926 07:21:07.433517 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa: Bootstrap starting.
I20260926 07:21:07.436906 8399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa: Neither blocks nor log segments found. Creating new log.
I20260926 07:21:07.437963 8399 log.cc:824] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:21:07 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:21:07.440624 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa: No bootstrap required, opened a new log
I20260926 07:21:07.445935 8399 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7d657eed62472d9d36f8bef3adccfa" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39199 } }
I20260926 07:21:07.446259 8399 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:21:07.446367 8399 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7d657eed62472d9d36f8bef3adccfa, State: Initialized, Role: FOLLOWER
I20260926 07:21:07.446966 8399 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7d657eed62472d9d36f8bef3adccfa" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39199 } }
I20260926 07:21:07.447172 8399 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:21:07.447276 8399 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:21:07.447386 8399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:21:07.448621 8399 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7d657eed62472d9d36f8bef3adccfa" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39199 } }
I20260926 07:21:07.448772 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:07.431921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.28.235.254:39199)
0926 07:21:07.432394 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:07.432414 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:07.432460 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:21:07.433188 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:07.433209 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:07.433603 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:07.434041 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:21:07.434889 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:07.435937 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:07.436873 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:07.437540 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:07.437740 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:07.441362 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:21:07.441369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:21:07.441389 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:21:07.441426 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:21:07.444603 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:07.445294 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:07.445300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:07.445308 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:07.445445 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:07.445782 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:07.445785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:07.445787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:07.446345 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:21:07.447206 (+ 861us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:07.447373 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 07:21:07.447714 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":159,"threads_started":1}
I20260926 07:21:07.449039 8399 leader_election.cc:302] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc7d657eed62472d9d36f8bef3adccfa; no voters:
I20260926 07:21:07.449460 8399 leader_election.cc:288] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:21:07.449530 8404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:21:07.449787 8404 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [term 1 LEADER]: Becoming Leader. State: Replica: fc7d657eed62472d9d36f8bef3adccfa, State: Running, Role: LEADER
I20260926 07:21:07.450141 8404 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7d657eed62472d9d36f8bef3adccfa" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39199 } }
I20260926 07:21:07.451035 8399 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:21:07.451444 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc7d657eed62472d9d36f8bef3adccfa. Latest consensus state: current_term: 1 leader_uuid: "fc7d657eed62472d9d36f8bef3adccfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7d657eed62472d9d36f8bef3adccfa" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39199 } } }
I20260926 07:21:07.451601 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [sys.catalog]: This master's current role is: LEADER
I20260926 07:21:07.451838 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc7d657eed62472d9d36f8bef3adccfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7d657eed62472d9d36f8bef3adccfa" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39199 } } }
I20260926 07:21:07.451952 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa [sys.catalog]: This master's current role is: LEADER
I20260926 07:21:07.452323 8408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:21:07.456269 8408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:21:07.456823 8399 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:21:07.470383 8408 catalog_manager.cc:1347] Generated new cluster ID: 1f74d70233a8404c87eb8e4e617ecd78
I20260926 07:21:07.470479 8408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:21:07.482640 8408 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:21:07.484216 8408 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:21:07.493515 8408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa: Generated new TSK 0
I20260926 07:21:07.494763 8408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:21:08.238198 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:52903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39199
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:21:08.400193 8444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:21:08.400595 8444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:21:08.400645 8444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:21:08.406436 8444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:21:08.406606 8444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:21:08.406785 8444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:21:08.415737 8444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.235.212:52903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:21:08.417303 8444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:21:08.418789 8444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:21:08.433101 8452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:08.433581 8454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:08.433775 8451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:21:08.435354 8444 server_base.cc:1034] running on GCE node
I20260926 07:21:08.435912 8444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:21:08.436851 8444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:21:08.438064 8444 hybrid_clock.cc:648] HybridClock initialized: now 1790407268438054 us; error 38 us; skew 500 ppm
Sep 26 07:21:08 dist-test-slave-7zlw krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:21:08.442073 8444 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:21:08.443322 8444 webserver.cc:460] Webserver started at http://127.28.235.193:45691/ using document root <none> and password file <none>
I20260926 07:21:08.444093 8444 fs_manager.cc:359] Metadata directory not provided
I20260926 07:21:08.444192 8444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:21:08.444583 8444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:21:08 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:21:08.462731 8444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "8c4fe4fd8586476fbfbe48f66cc8643f"
format_stamp: "Formatted at 2026-09-26 07:21:08 on dist-test-slave-7zlw"
server_key: "d24a962abdaa9f106c4da460c949cda6"
server_key_iv: "88be9ebb78bde0496761273307d51149"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:08.463501 8444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "8c4fe4fd8586476fbfbe48f66cc8643f"
format_stamp: "Formatted at 2026-09-26 07:21:08 on dist-test-slave-7zlw"
server_key: "d24a962abdaa9f106c4da460c949cda6"
server_key_iv: "88be9ebb78bde0496761273307d51149"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:08.475636 8444 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.000s
I20260926 07:21:08.513517 8463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:08.515323 8444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:21:08.515491 8444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "8c4fe4fd8586476fbfbe48f66cc8643f"
format_stamp: "Formatted at 2026-09-26 07:21:08 on dist-test-slave-7zlw"
server_key: "d24a962abdaa9f106c4da460c949cda6"
server_key_iv: "88be9ebb78bde0496761273307d51149"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:08.515632 8444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:21:08.541230 8444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:21:08.545316 8444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:21:08.545584 8444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:21:08.546434 8444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:21:08.547633 8444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:21:08.547735 8444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:08.547829 8444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:21:08.547881 8444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:08.559571 8444 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:37403
I20260926 07:21:08.559636 8576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:37403 every 8 connection(s)
I20260926 07:21:08.561064 8444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:21:08.568756 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 8444
I20260926 07:21:08.568900 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:21:08.569463 29615 external_mini_cluster.cc:1428] Setting key f860bc009780b53a46678e4ae363e78c
Sep 26 07:21:08 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:21:08.584292 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:08.565148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:57997 (local address 127.28.235.254:39199)
0926 07:21:08.565406 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:08.565411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:08.565430 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:21:08.566053 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:08.566056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:08.566117 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:08.566224 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:21:08.567791 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:08.568631 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:08.570844 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:08.571472 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:08.571595 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:08.576436 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:21:08.576442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:21:08.576446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:21:08.576488 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:21:08.578853 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:08.581194 (+ 2341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:08.581199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:08.581201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:08.581334 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:08.582000 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:08.582004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:08.582006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:08.582297 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:21:08.582400 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:08.583913 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0926 07:21:08.584088 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":65,"threads_started":1}
I20260926 07:21:08.586728 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:08.563904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39199 (local address 127.28.235.193:57997)
0926 07:21:08.564563 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:08.564629 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:08.565758 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:08.566461 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:08.566471 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:08.566854 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:21:08.567578 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:08.567599 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:08.568800 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:08.568804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:08.570675 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:08.570691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:08.571800 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:08.571804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:08.572050 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:08.572830 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:21:08.572854 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:21:08.576103 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:21:08.579056 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:08.579065 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:08.579090 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:08.579566 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:08.581485 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:08.581488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:08.581490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:08.581753 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:08.582827 (+ 1074us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:21:08.582837 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:21:08.583557 (+ 720us) client_negotiation.cc:769] Sending connection context
0926 07:21:08.585040 (+ 1483us) client_negotiation.cc:240] Negotiation successful
0926 07:21:08.585498 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":158,"threads_started":1}
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:21:08.588527 8577 heartbeater.cc:347] Connected to a master server at 127.28.235.254:39199
I20260926 07:21:08.588832 8577 heartbeater.cc:464] Registering TS with master...
I20260926 07:21:08.589617 8577 heartbeater.cc:511] Master 127.28.235.254:39199 requested a full tablet report, sending...
I20260926 07:21:08.591799 8363 ts_manager.cc:194] Registered new tserver with Master: 8c4fe4fd8586476fbfbe48f66cc8643f (127.28.235.193:37403)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:21:08.593532 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:57997
Sep 26 07:21:08 dist-test-slave-7zlw krb5kdc[7887](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:21:08.617259 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:52903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39199
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:21:08.776319 8583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:21:08.776736 8583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:21:08.776799 8583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:21:08.783356 8583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:21:08.783458 8583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:21:08.783546 8583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:21:08.792301 8583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.235.212:52903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:21:08.794030 8583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:21:08.795521 8583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:21:08.809787 8592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:08.810351 8590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:08.811156 8589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:21:08.812457 8583 server_base.cc:1034] running on GCE node
I20260926 07:21:08.813140 8583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:21:08.814385 8583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:21:08.815591 8583 hybrid_clock.cc:648] HybridClock initialized: now 1790407268815556 us; error 59 us; skew 500 ppm
Sep 26 07:21:08 dist-test-slave-7zlw krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:21:08.819841 8583 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:21:08.821377 8583 webserver.cc:460] Webserver started at http://127.28.235.194:35505/ using document root <none> and password file <none>
I20260926 07:21:08.822270 8583 fs_manager.cc:359] Metadata directory not provided
I20260926 07:21:08.822446 8583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:21:08.822775 8583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:21:08 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:21:08.842664 8583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "f7b4fa53e06f417aa579c7fc8fc48459"
format_stamp: "Formatted at 2026-09-26 07:21:08 on dist-test-slave-7zlw"
server_key: "2430f39b7850ba72a63a42f3d531235a"
server_key_iv: "00a4709ea7f699dbef9522bb39bb49ef"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:08.843665 8583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "f7b4fa53e06f417aa579c7fc8fc48459"
format_stamp: "Formatted at 2026-09-26 07:21:08 on dist-test-slave-7zlw"
server_key: "2430f39b7850ba72a63a42f3d531235a"
server_key_iv: "00a4709ea7f699dbef9522bb39bb49ef"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:08.849701 8583 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 07:21:08.869402 8599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:08.871069 8583 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 07:21:08.871232 8583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "f7b4fa53e06f417aa579c7fc8fc48459"
format_stamp: "Formatted at 2026-09-26 07:21:08 on dist-test-slave-7zlw"
server_key: "2430f39b7850ba72a63a42f3d531235a"
server_key_iv: "00a4709ea7f699dbef9522bb39bb49ef"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:08.871398 8583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:21:08.891989 8583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:21:08.896543 8583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:21:08.896833 8583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:21:08.897830 8583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:21:08.899132 8583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:21:08.899268 8583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:08.899377 8583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:21:08.899444 8583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:08.917881 8583 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:43475
I20260926 07:21:08.918998 8583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:21:08.925238 8712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:43475 every 8 connection(s)
I20260926 07:21:08.928462 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 8583
I20260926 07:21:08.928601 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:21:08.928956 29615 external_mini_cluster.cc:1428] Setting key 0e1ad9b1527a90588c1068d9ff1b0970
Sep 26 07:21:08 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:21:08.940492 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:08.921488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:58669 (local address 127.28.235.254:39199)
0926 07:21:08.921625 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:08.921630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:08.922980 (+ 1350us) server_negotiation.cc:407] Connection header received
0926 07:21:08.923990 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:08.923994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:08.924055 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:08.924129 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:21:08.927307 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:08.928185 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:08.931283 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:08.931655 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:08.931740 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:08.935671 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:21:08.935675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:21:08.935678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:21:08.935711 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:21:08.937712 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:08.938430 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:08.938434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:08.938436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:08.938546 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:08.939031 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:08.939035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:08.939037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:08.939277 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:21:08.939373 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:08.940150 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 07:21:08.940308 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:21:08.942798 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:08.922069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39199 (local address 127.28.235.194:58669)
0926 07:21:08.922793 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:08.922862 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:08.923745 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:08.925898 (+ 2153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:08.925909 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:08.926295 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:21:08.927058 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:08.927081 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:08.928899 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:08.928903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:08.931123 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:08.931138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:08.932368 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:08.932372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:08.932618 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:08.933397 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:21:08.933420 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:21:08.935427 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:21:08.937898 (+ 2471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:08.937905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:08.937927 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:08.938303 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:08.938704 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:08.938706 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:08.938708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:08.938913 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:08.939445 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:21:08.939452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:21:08.939830 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 07:21:08.941155 (+ 1325us) client_negotiation.cc:240] Negotiation successful
0926 07:21:08.941572 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":166,"threads_started":1}
I20260926 07:21:08.944379 8713 heartbeater.cc:347] Connected to a master server at 127.28.235.254:39199
I20260926 07:21:08.944650 8713 heartbeater.cc:464] Registering TS with master...
I20260926 07:21:08.945544 8713 heartbeater.cc:511] Master 127.28.235.254:39199 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
I20260926 07:21:08.946791 8363 ts_manager.cc:194] Registered new tserver with Master: f7b4fa53e06f417aa579c7fc8fc48459 (127.28.235.194:43475)
I20260926 07:21:08.947607 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:58669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:21:08.970559 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:52903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39199
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:21:09 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
W20260926 07:21:09.143883 8718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:21:09.144232 8718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:21:09.144318 8718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:21:09.148841 8718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:21:09.148955 8718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:21:09.149080 8718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:21:09.155409 8718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.235.212:52903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:21:09.156643 8718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:21:09.157900 8718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:21:09.168785 8726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:09.168905 8723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:09.168785 8724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:21:09.170032 8718 server_base.cc:1034] running on GCE node
I20260926 07:21:09.170532 8718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:21:09.171427 8718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:21:09.172704 8718 hybrid_clock.cc:648] HybridClock initialized: now 1790407269172619 us; error 104 us; skew 500 ppm
Sep 26 07:21:09 dist-test-slave-7zlw krb5kdc[7887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407269, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:21:09.176067 8718 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:21:09.177213 8718 webserver.cc:460] Webserver started at http://127.28.235.195:45829/ using document root <none> and password file <none>
I20260926 07:21:09.177860 8718 fs_manager.cc:359] Metadata directory not provided
I20260926 07:21:09.177961 8718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:21:09.178225 8718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:21:09 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407269, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:21:09.193646 8718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "957fe23327bf4e6d92a072adee19be3e"
format_stamp: "Formatted at 2026-09-26 07:21:09 on dist-test-slave-7zlw"
server_key: "00a97c3a0223f1e7b89db7e80397bacf"
server_key_iv: "855caf8db1059f4eac4cdca215b3019d"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:09.194581 8718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "957fe23327bf4e6d92a072adee19be3e"
format_stamp: "Formatted at 2026-09-26 07:21:09 on dist-test-slave-7zlw"
server_key: "00a97c3a0223f1e7b89db7e80397bacf"
server_key_iv: "855caf8db1059f4eac4cdca215b3019d"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:09.206007 8718 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260926 07:21:09.224450 8733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:09.226325 8718 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:21:09.226498 8718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "957fe23327bf4e6d92a072adee19be3e"
format_stamp: "Formatted at 2026-09-26 07:21:09 on dist-test-slave-7zlw"
server_key: "00a97c3a0223f1e7b89db7e80397bacf"
server_key_iv: "855caf8db1059f4eac4cdca215b3019d"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:09.226653 8718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:21:09.249145 8718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:21:09.253489 8718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:21:09.253697 8718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:21:09.254456 8718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:21:09.255584 8718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:21:09.255654 8718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:09.255724 8718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:21:09.255755 8718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:09.267279 8718 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:33513
I20260926 07:21:09.267333 8846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:33513 every 8 connection(s)
I20260926 07:21:09.268559 8718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:21:09.273231 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 8718
I20260926 07:21:09.273365 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:21:09.273648 29615 external_mini_cluster.cc:1428] Setting key 2a8356102809dbcd92b79dc229bd90e5
Sep 26 07:21:09 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407269, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:21:09.287416 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.272524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:57961 (local address 127.28.235.254:39199)
0926 07:21:09.272673 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:09.272678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:09.272695 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:21:09.273698 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:09.273702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:09.273766 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:09.273832 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:21:09.275771 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.276688 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.278636 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.278997 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.279111 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.282388 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:21:09.282393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:21:09.282396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:21:09.282430 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:21:09.284315 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:09.285065 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:09.285070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:09.285072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:09.285186 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:09.285667 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:09.285673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:09.285676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:09.285948 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:21:09.286067 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:09.287091 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0926 07:21:09.287248 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:21:09.289108 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.271351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39199 (local address 127.28.235.195:57961)
0926 07:21:09.271977 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:09.272044 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:09.273409 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:09.274398 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:09.274409 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:09.274793 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:21:09.275590 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.275612 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.276911 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.276915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.278471 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.278486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.279198 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.279203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.279443 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.280242 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:21:09.280266 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:21:09.282136 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:21:09.284465 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:09.284473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:09.284495 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:09.284889 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:09.285296 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:09.285299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:09.285301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:09.285529 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:09.286376 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:21:09.286384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:21:09.286758 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 07:21:09.287576 (+ 818us) client_negotiation.cc:240] Negotiation successful
0926 07:21:09.287977 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":150,"threads_started":1}
I20260926 07:21:09.290373 8847 heartbeater.cc:347] Connected to a master server at 127.28.235.254:39199
I20260926 07:21:09.290653 8847 heartbeater.cc:464] Registering TS with master...
I20260926 07:21:09.291246 8847 heartbeater.cc:511] Master 127.28.235.254:39199 requested a full tablet report, sending...
I20260926 07:21:09.292485 8363 ts_manager.cc:194] Registered new tserver with Master: 957fe23327bf4e6d92a072adee19be3e (127.28.235.195:33513)
I20260926 07:21:09.293314 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:57961
I20260926 07:21:09.300694 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:21:09.313743 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.303137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.28.235.254:39199)
0926 07:21:09.303266 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:09.303271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:09.303721 (+ 450us) server_negotiation.cc:407] Connection header received
0926 07:21:09.303891 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:09.303895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:09.303951 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:09.304004 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:21:09.305154 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.306014 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.307343 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.307675 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.307785 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.309712 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:21:09.309717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:21:09.309720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:21:09.309756 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:21:09.311562 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:09.312119 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:09.312123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:09.312124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:09.312225 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:09.312557 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:09.312560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:09.312562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:09.312820 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:21:09.312960 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:09.313434 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 07:21:09.313571 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:21:09.317530 8363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:21:09.321892 8363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:21:09.343379 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.334083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:37403 (local address 127.0.0.1:36718)
0926 07:21:09.336405 (+ 2322us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:09.336419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:09.336507 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:09.337457 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:09.337465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:09.337491 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:21:09.337648 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.337656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.339079 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.339083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.341404 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.341415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.342809 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.342813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.342926 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.342965 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:21:09.343076 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:21:09.343207 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2254,"thread_start_us":98,"threads_started":1}
I20260926 07:21:09.343871 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.334754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.28.235.193:37403)
0926 07:21:09.335183 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:09.335191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:09.336612 (+ 1421us) server_negotiation.cc:407] Connection header received
0926 07:21:09.336701 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:09.336707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:09.336963 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:09.337167 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:21:09.337787 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.338939 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.341549 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.342469 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.342570 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.342706 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:09.343615 (+ 909us) server_negotiation.cc:299] Negotiation successful
0926 07:21:09.343732 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":125,"threads_started":1}
I20260926 07:21:09.345884 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.333219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:43475 (local address 127.0.0.1:56420)
0926 07:21:09.335943 (+ 2724us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:09.336008 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:09.336150 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:09.339951 (+ 3801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:09.339954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:09.339974 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:21:09.340081 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.340087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.341510 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.341514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.342670 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.342683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.345417 (+ 2734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.345421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.345537 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.345568 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:21:09.345644 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 07:21:09.345732 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2518,"thread_start_us":123,"threads_started":1}
I20260926 07:21:09.346930 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.333489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.28.235.194:43475)
0926 07:21:09.339259 (+ 5770us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:09.339273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:09.339299 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:21:09.339396 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:09.339402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:09.339638 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:09.339803 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:21:09.340207 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.341361 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.342879 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.344091 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.344221 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.346609 (+ 2388us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:09.346671 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 07:21:09.346781 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5644,"thread_start_us":136,"threads_started":1}
I20260926 07:21:09.348214 8509 tablet_service.cc:1467] Processing CreateTablet for tablet 3080e576d6534cb5ba830b2b8fdfb260 (DEFAULT_TABLE table=test-table [id=686c1dce0c9a426d8ad54ee30d484fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:21:09.350145 8509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3080e576d6534cb5ba830b2b8fdfb260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:21:09.350559 8647 tablet_service.cc:1467] Processing CreateTablet for tablet 3080e576d6534cb5ba830b2b8fdfb260 (DEFAULT_TABLE table=test-table [id=686c1dce0c9a426d8ad54ee30d484fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:21:09.352929 8647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3080e576d6534cb5ba830b2b8fdfb260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:21:09.354553 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.334793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:33513 (local address 127.0.0.1:60624)
0926 07:21:09.337179 (+ 2386us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:09.337194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:09.337288 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:09.349905 (+ 12617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:09.349909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:09.349934 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:21:09.350044 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.350051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.351467 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.351471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.352633 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.352646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.354098 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.354102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.354197 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.354227 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:21:09.354307 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:21:09.354400 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2304,"thread_start_us":95,"threads_started":1}
I20260926 07:21:09.355345 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.335206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.28.235.195:33513)
0926 07:21:09.335562 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:09.335573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:09.349097 (+ 13524us) server_negotiation.cc:407] Connection header received
0926 07:21:09.349200 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:09.349205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:09.349478 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:09.349683 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:21:09.350188 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.351304 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.352820 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.353675 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.353811 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.354018 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:09.355077 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0926 07:21:09.355191 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260926 07:21:09.358798 8776 tablet_service.cc:1467] Processing CreateTablet for tablet 3080e576d6534cb5ba830b2b8fdfb260 (DEFAULT_TABLE table=test-table [id=686c1dce0c9a426d8ad54ee30d484fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:21:09.360050 8776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3080e576d6534cb5ba830b2b8fdfb260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:21:09.373131 8865 tablet_bootstrap.cc:492] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: Bootstrap starting.
I20260926 07:21:09.375149 8867 tablet_bootstrap.cc:492] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: Bootstrap starting.
I20260926 07:21:09.375540 8865 tablet_bootstrap.cc:654] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: Neither blocks nor log segments found. Creating new log.
I20260926 07:21:09.377063 8865 log.cc:824] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: Log is configured to *not* fsync() on all Append() calls
I20260926 07:21:09.378160 8867 tablet_bootstrap.cc:654] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: Neither blocks nor log segments found. Creating new log.
I20260926 07:21:09.379021 8867 log.cc:824] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:21:09.381579 8867 tablet_bootstrap.cc:492] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: No bootstrap required, opened a new log
I20260926 07:21:09.381829 8867 ts_tablet_manager.cc:1397] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:21:09.385437 8866 tablet_bootstrap.cc:492] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f: Bootstrap starting.
I20260926 07:21:09.385970 8865 tablet_bootstrap.cc:492] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: No bootstrap required, opened a new log
I20260926 07:21:09.386214 8865 ts_tablet_manager.cc:1397] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260926 07:21:09.386883 8867 raft_consensus.cc:348] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 } } peers { permanent_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 37403 } } peers { permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 } }
I20260926 07:21:09.387193 8867 raft_consensus.cc:374] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:21:09.387254 8867 raft_consensus.cc:729] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957fe23327bf4e6d92a072adee19be3e, State: Initialized, Role: FOLLOWER
I20260926 07:21:09.387770 8866 tablet_bootstrap.cc:654] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f: Neither blocks nor log segments found. Creating new log.
I20260926 07:21:09.387784 8867 consensus_queue.cc:260] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 } } peers { permanent_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 37403 } } peers { permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 } }
I20260926 07:21:09.388547 8847 heartbeater.cc:503] Master 127.28.235.254:39199 was elected leader, sending a full tablet report...
I20260926 07:21:09.388720 8866 log.cc:824] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:21:09.388913 8867 ts_tablet_manager.cc:1428] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:21:09.391103 8865 raft_consensus.cc:348] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 } } peers { permanent_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 37403 } } peers { permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 } }
I20260926 07:21:09.391390 8865 raft_consensus.cc:374] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:21:09.391448 8865 raft_consensus.cc:729] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b4fa53e06f417aa579c7fc8fc48459, State: Initialized, Role: FOLLOWER
I20260926 07:21:09.391952 8865 consensus_queue.cc:260] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 } } peers { permanent_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 37403 } } peers { permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 } }
I20260926 07:21:09.393071 8865 ts_tablet_manager.cc:1428] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:21:09.393740 8713 heartbeater.cc:503] Master 127.28.235.254:39199 was elected leader, sending a full tablet report...
I20260926 07:21:09.395151 8866 tablet_bootstrap.cc:492] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f: No bootstrap required, opened a new log
I20260926 07:21:09.395383 8866 ts_tablet_manager.cc:1397] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260926 07:21:09.400291 8866 raft_consensus.cc:348] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 } } peers { permanent_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 37403 } } peers { permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 } }
I20260926 07:21:09.400590 8866 raft_consensus.cc:374] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:21:09.400648 8866 raft_consensus.cc:729] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4fe4fd8586476fbfbe48f66cc8643f, State: Initialized, Role: FOLLOWER
I20260926 07:21:09.401222 8866 consensus_queue.cc:260] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 } } peers { permanent_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 37403 } } peers { permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 } }
I20260926 07:21:09.402488 8866 ts_tablet_manager.cc:1428] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260926 07:21:09.402911 8577 heartbeater.cc:503] Master 127.28.235.254:39199 was elected leader, sending a full tablet report...
Sep 26 07:21:09 dist-test-slave-7zlw krb5kdc[7887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407268, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
W20260926 07:21:09.421792 8714 tablet.cc:2369] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:21:09.443629 8874 raft_consensus.cc:482] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:21:09.443883 8874 raft_consensus.cc:504] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 } } peers { permanent_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 37403 } } peers { permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 } }
I20260926 07:21:09.445371 8874 leader_election.cc:288] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7b4fa53e06f417aa579c7fc8fc48459 (127.28.235.194:43475), 957fe23327bf4e6d92a072adee19be3e (127.28.235.195:33513)
I20260926 07:21:09.452153 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.446906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:34801 (local address 127.28.235.194:43475)
0926 07:21:09.447163 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:09.447168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:09.447517 (+ 349us) server_negotiation.cc:407] Connection header received
0926 07:21:09.447732 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:09.447736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:09.447788 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:09.447867 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:21:09.448304 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.449071 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.450718 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.451336 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.451455 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.451516 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:09.451915 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 07:21:09.452004 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260926 07:21:09.452816 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.445937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:43475 (local address 127.28.235.193:34801)
0926 07:21:09.447429 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:09.447444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:09.447602 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:09.448017 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:09.448020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:09.448036 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:21:09.448176 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.448184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.449191 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.449196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.450566 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.450577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.451643 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.451647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.451758 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.451786 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:21:09.452557 (+ 771us) client_negotiation.cc:240] Negotiation successful
0926 07:21:09.452645 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"thread_start_us":148,"threads_started":1}
I20260926 07:21:09.453679 8661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3080e576d6534cb5ba830b2b8fdfb260" candidate_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" is_pre_election: true
I20260926 07:21:09.454037 8661 raft_consensus.cc:2454] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c4fe4fd8586476fbfbe48f66cc8643f in term 0.
I20260926 07:21:09.454687 8465 leader_election.cc:302] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c4fe4fd8586476fbfbe48f66cc8643f, f7b4fa53e06f417aa579c7fc8fc48459; no voters:
I20260926 07:21:09.455045 8874 raft_consensus.cc:2781] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:21:09.455142 8874 raft_consensus.cc:482] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:21:09.455199 8874 raft_consensus.cc:3037] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:21:09.456645 8874 raft_consensus.cc:504] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 } } peers { permanent_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 37403 } } peers { permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 } }
I20260926 07:21:09.458465 8874 leader_election.cc:288] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [CANDIDATE]: Term 1 election: Requested vote from peers f7b4fa53e06f417aa579c7fc8fc48459 (127.28.235.194:43475), 957fe23327bf4e6d92a072adee19be3e (127.28.235.195:33513)
I20260926 07:21:09.458972 8661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3080e576d6534cb5ba830b2b8fdfb260" candidate_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b4fa53e06f417aa579c7fc8fc48459"
I20260926 07:21:09.459126 8661 raft_consensus.cc:3037] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:21:09.460512 8661 raft_consensus.cc:2454] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c4fe4fd8586476fbfbe48f66cc8643f in term 1.
I20260926 07:21:09.460980 8465 leader_election.cc:302] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c4fe4fd8586476fbfbe48f66cc8643f, f7b4fa53e06f417aa579c7fc8fc48459; no voters:
I20260926 07:21:09.461252 8874 raft_consensus.cc:2781] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:21:09.461536 8874 raft_consensus.cc:686] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [term 1 LEADER]: Becoming Leader. State: Replica: 8c4fe4fd8586476fbfbe48f66cc8643f, State: Running, Role: LEADER
I20260926 07:21:09.461949 8874 consensus_queue.cc:237] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 } } peers { permanent_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 37403 } } peers { permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 } }
I20260926 07:21:09.466645 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.446425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:33513 (local address 127.28.235.193:50863)
0926 07:21:09.448623 (+ 2198us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:09.448638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:09.448774 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:09.449155 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:09.449158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:09.449175 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:21:09.449292 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.449297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.453248 (+ 3951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.453251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.454321 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:09.454331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.466100 (+ 11769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.466104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:09.466214 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.466243 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:21:09.466367 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:21:09.466461 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2131,"thread_start_us":99,"threads_started":1}
I20260926 07:21:09.466681 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.447032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:50863 (local address 127.28.235.195:33513)
0926 07:21:09.447250 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:09.447254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:09.448713 (+ 1459us) server_negotiation.cc:407] Connection header received
0926 07:21:09.448897 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:09.448900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:09.448948 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:09.448993 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:21:09.452306 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.453052 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.465176 (+ 12124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.465812 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.465941 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.465996 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:09.466434 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 07:21:09.466538 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260926 07:21:09.466687 8363 catalog_manager.cc:5495] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c4fe4fd8586476fbfbe48f66cc8643f (127.28.235.193). New cstate: current_term: 1 leader_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 37403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 } health_report { overall_health: UNKNOWN } } }
I20260926 07:21:09.469964 8800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3080e576d6534cb5ba830b2b8fdfb260" candidate_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957fe23327bf4e6d92a072adee19be3e" is_pre_election: true
I20260926 07:21:09.470316 8800 raft_consensus.cc:2454] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c4fe4fd8586476fbfbe48f66cc8643f in term 0.
I20260926 07:21:09.470690 8783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3080e576d6534cb5ba830b2b8fdfb260" candidate_uuid: "8c4fe4fd8586476fbfbe48f66cc8643f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957fe23327bf4e6d92a072adee19be3e"
I20260926 07:21:09.470805 8783 raft_consensus.cc:3037] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:21:09.472198 8783 raft_consensus.cc:2454] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c4fe4fd8586476fbfbe48f66cc8643f in term 1.
I20260926 07:21:09.498103 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.493664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.28.235.193:37403)
0926 07:21:09.493789 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:09.493794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:09.493808 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:21:09.493913 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:09.493917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:09.493991 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:09.494103 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:21:09.494506 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.495342 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.496465 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:09.496722 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:09.496841 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:09.497217 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:21:09.497675 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:21:09.497824 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:09.497877 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 07:21:09.497942 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:21:09.505772 8783 raft_consensus.cc:1261] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e [term 1 FOLLOWER]: Refusing update from remote peer 8c4fe4fd8586476fbfbe48f66cc8643f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:21:09.506980 8878 consensus_queue.cc:1035] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [LEADER]: Connected to new peer: Peer: permanent_uuid: "957fe23327bf4e6d92a072adee19be3e" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 33513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:21:09.508060 8661 raft_consensus.cc:1261] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459 [term 1 FOLLOWER]: Refusing update from remote peer 8c4fe4fd8586476fbfbe48f66cc8643f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:21:09.508708 8874 consensus_queue.cc:1035] T 3080e576d6534cb5ba830b2b8fdfb260 P 8c4fe4fd8586476fbfbe48f66cc8643f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7b4fa53e06f417aa579c7fc8fc48459" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 43475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:21:09.519915 8848 tablet.cc:2369] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:21:09.534132 8881 mvcc.cc:205] Tried to move back new op lower bound from 7333508175886835712 to 7333508175722491904. Current Snapshot: MvccSnapshot[applied={T|T < 7333508175886835712}]
I20260926 07:21:09.542722 8884 mvcc.cc:205] Tried to move back new op lower bound from 7333508175886835712 to 7333508175722491904. Current Snapshot: MvccSnapshot[applied={T|T < 7333508175886835712}]
I20260926 07:21:09.559962 8363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:21:09.560174 8363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:21:09.563720 8363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc7d657eed62472d9d36f8bef3adccfa: Sending DeleteTablet for 3 replicas of tablet 3080e576d6534cb5ba830b2b8fdfb260
I20260926 07:21:09.564991 8509 tablet_service.cc:1514] Processing DeleteTablet for tablet 3080e576d6534cb5ba830b2b8fdfb260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:21:09 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:36718
I20260926 07:21:09.565132 8647 tablet_service.cc:1514] Processing DeleteTablet for tablet 3080e576d6534cb5ba830b2b8fdfb260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:21:09 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:56420
I20260926 07:21:09.565589 8776 tablet_service.cc:1514] Processing DeleteTablet for tablet 3080e576d6534cb5ba830b2b8fdfb260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:21:09 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:60624
I20260926 07:21:09.567060 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 8444
I20260926 07:21:09.577332 8891 tablet_replica.cc:330] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: stopping tablet replica
I20260926 07:21:09.577811 8891 raft_consensus.cc:2229] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:21:09.578238 8891 raft_consensus.cc:2258] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:21:09.579306 8892 tablet_replica.cc:330] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: stopping tablet replica
I20260926 07:21:09.579674 8892 raft_consensus.cc:2229] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:21:09.580066 8892 raft_consensus.cc:2258] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:21:09.580785 8891 ts_tablet_manager.cc:1905] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:21:09.581894 8892 ts_tablet_manager.cc:1905] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:21:09.584515 8891 ts_tablet_manager.cc:1918] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:21:09.584724 8891 log.cc:1196] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/3080e576d6534cb5ba830b2b8fdfb260
I20260926 07:21:09.584810 8892 ts_tablet_manager.cc:1918] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:21:09.584921 8892 log.cc:1196] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/3080e576d6534cb5ba830b2b8fdfb260
I20260926 07:21:09.585322 8892 ts_tablet_manager.cc:1939] T 3080e576d6534cb5ba830b2b8fdfb260 P 957fe23327bf4e6d92a072adee19be3e: Deleting consensus metadata
I20260926 07:21:09.585322 8891 ts_tablet_manager.cc:1939] T 3080e576d6534cb5ba830b2b8fdfb260 P f7b4fa53e06f417aa579c7fc8fc48459: Deleting consensus metadata
I20260926 07:21:09.586845 8349 catalog_manager.cc:4841] TS 957fe23327bf4e6d92a072adee19be3e (127.28.235.195:33513): tablet 3080e576d6534cb5ba830b2b8fdfb260 (table test-table [id=686c1dce0c9a426d8ad54ee30d484fee]) successfully deleted
I20260926 07:21:09.586848 8348 catalog_manager.cc:4841] TS f7b4fa53e06f417aa579c7fc8fc48459 (127.28.235.194:43475): tablet 3080e576d6534cb5ba830b2b8fdfb260 (table test-table [id=686c1dce0c9a426d8ad54ee30d484fee]) successfully deleted
W20260926 07:21:09.597957 8347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:37403 (error 108)
I20260926 07:21:09.599323 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 8583
I20260926 07:21:09.601075 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:09.600588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:37403 (local address 127.0.0.1:36722)
0926 07:21:09.600823 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:09.600913 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:37403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260926 07:21:09.601472 8347 catalog_manager.cc:4584] TS 8c4fe4fd8586476fbfbe48f66cc8643f (127.28.235.193:37403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3080e576d6534cb5ba830b2b8fdfb260: Network error: Client connection negotiation failed: client connection to 127.28.235.193:37403: connect: Connection refused (error 111)
I20260926 07:21:09.607573 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 8718
I20260926 07:21:09.615278 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 8331
I20260926 07:21:09.623688 29615 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:21:09.969861 29615 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:21:09.970036 29615 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:21:10.316128 29615 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:21:10.316 UTC [7917] LOG: received smart shutdown request
2026-09-26 07:21:10.320 UTC [7917] LOG: background worker "logical replication launcher" (PID 7926) exited with exit code 1
2026-09-26 07:21:10.320 UTC [7921] LOG: shutting down
2026-09-26 07:21:10.548 UTC [7917] LOG: database system is shut down
2026-09-26T07:21:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49527 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:21:10 dist-test-slave-7zlw krb5kdc[8902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:21:10 dist-test-slave-7zlw krb5kdc[8902](info): set up 2 sockets
Sep 26 07:21:10 dist-test-slave-7zlw krb5kdc[8902](info): commencing operation
krb5kdc: starting...
W20260926 07:21:11.741299 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:21:11 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:21:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/rangeradmin_127.28.235.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/rangeradmin_127.28.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/rangerlookup_127.28.235.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/rangerlookup_127.28.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/HTTP_127.28.235.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/HTTP_127.28.235.212@KRBTEST.COM.keytab.
I20260926 07:21:11.946573 29615 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/postgres -l logfile start
2026-09-26 07:21:15.098 UTC [8932] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:21:15.099 UTC [8932] LOG: listening on IPv4 address "127.28.235.212", port 55269
2026-09-26 07:21:15.104 UTC [8932] LOG: listening on Unix socket "/tmp/.s.PGSQL.55269"
2026-09-26 07:21:15.130 UTC [8935] LOG: database system was shut down at 2026-09-26 07:21:13 UTC
2026-09-26 07:21:15.135 UTC [8932] LOG: database system is ready to accept connections
I20260926 07:21:16.117153 29615 mini_postgres.cc:96] Postgres bound to 55269
2026-09-26 07:21:16.123 UTC [8943] FATAL: database "slave" does not exist
127.28.235.212:55269 - accepting connections
I20260926 07:21:16.124500 29615 mini_ranger.cc:161] Starting Ranger...
I20260926 07:21:16.141076 29615 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:21:16.859572 29615 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:21:16.859671 29615 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-admin
2026-09-26 07:21:17,177 [I] DB FLAVOR :POSTGRES
2026-09-26 07:21:17,178 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:21:17,178 [I] Checking connection..
2026-09-26 07:21:17,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:21:17,444 [I] Checking connection passed.
2026-09-26 07:21:17,445 [I] --------- Verifying version history table ---------
2026-09-26 07:21:17,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:21:17,694 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:21:17,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:21:17,958 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:21:17,958 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:21:17,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:21:18.222 UTC [9007] WARNING: there is no transaction in progress
2026-09-26 07:21:18,230 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:21:18,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:21:18,516 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:21:18,516 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:21:18,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:21:18,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:21:19,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7zlw.c.gcp-upstream.internal','N') ;"
2026-09-26 07:21:19,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:21:19,575 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:21:19,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:21:19,825 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:21:19,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:21:20,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:21:20,349 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:21:20,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:21:21.533 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.553 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.575 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.594 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.615 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.637 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.704 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.715 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.735 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.749 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.762 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.780 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.793 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.802 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.812 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:21.821 UTC [9115] WARNING: there is no transaction in progress
2026-09-26 07:21:22,745 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:21:22,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:21:23,018 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:21:23,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:21:23,301 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:21:23,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:21:23,565 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:21:23,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:21:23,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:21:24,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7zlw.c.gcp-upstream.internal';"
2026-09-26 07:21:24,365 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:21:24,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:21:24,635 [I] DB_PATCHES have already been applied
I20260926 07:21:24.641396 29615 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:21:24.641471 29615 mini_ranger.cc:193] Using host: 127.28.235.212
I20260926 07:21:24.645337 29615 mini_ranger.cc:227] Ranger admin URL: http://127.28.235.212:42163
Sep 26, 2026 7:21:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:21:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:21:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:21:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:21:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:21:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:21:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:21:25.787331 29615 mini_ranger.cc:160] Time spent starting Ranger: real 9.663s user 0.000s sys 0.011s
Sep 26, 2026 7:21:46 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:21:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 7:21:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:21:46 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:21:47 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:21:48.553952 29615 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/rangerkms_127.28.235.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/rangerkms_127.28.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.235.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.235.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.235.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/HTTP_127.28.235.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.235.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/HTTP_127.28.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:21:48 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407308, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:21:48.668910 29615 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:21:48.680291 29615 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:21:49.749531 29615 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:21:49.749622 29615 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-kms
I20260926 07:21:49.749645 29615 mini_ranger_kms.cc:227] Using postgres at 127.28.235.212:55269
2026-09-26 07:21:49,928 [I] DB FLAVOR :POSTGRES
2026-09-26 07:21:49,928 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:21:49,928 [I] Checking connection
2026-09-26 07:21:49,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:21:50,196 [I] connection success
2026-09-26 07:21:50,196 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:21:50,196 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:21:50,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:21:50,464 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:21:50,465 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:21:50,465 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:21:50,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.235.212:55269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:21:50,752 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:21:51.299620 29615 mini_ranger_kms.cc:310] Created kms service
I20260926 07:21:51.524376 29615 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:21:51.570094 29615 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:21:51 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407308, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:21:51.984514 29615 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:21:51.984669 29615 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZ0ZD4L/build/debug/bin/postgresql.jar:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZ0ZD4L/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:21:51.984694 29615 mini_ranger_kms.cc:240] Using host: 127.28.235.212
I20260926 07:21:51.988350 29615 mini_ranger_kms.cc:276] Ranger KMS PID: 9296
I20260926 07:21:51.988447 29615 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.235.212:45715
Sep 26, 2026 7:21:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:21:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:21:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:21:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:21:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:21:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 07:21:54.240768 29615 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.572s user 0.000s sys 0.161s
I20260926 07:21:54.240890 29615 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:21:54.240957 29615 mini_ranger_kms.cc:401] 127.28.235.212:45715/kms/v1/keys
Sep 26 07:21:54 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407314, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.235.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:21:54 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407314, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.235.212@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
Sep 26 07:21:55 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407314, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.235.212@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
Sep 26 07:21:57 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:21:57.553525 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:37009
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:37009
--ranger_config_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:45715/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:21:57.675084 9346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:21:57.675395 9346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:21:57.675436 9346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:21:57.680389 9346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:21:57.680473 9346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:21:57.680495 9346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:21:57.680512 9346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:21:57.680528 9346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:21:57.686734 9346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.235.212:45715/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:37009
--ranger_config_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:37009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:21:57.687934 9346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:21:57.689087 9346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:21:57.698962 9346 server_base.cc:1034] running on GCE node
W20260926 07:21:57.698868 9351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:57.698886 9352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:57.698863 9354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:21:57.699898 9346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:21:57.701321 9346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:21:57.702508 9346 hybrid_clock.cc:648] HybridClock initialized: now 1790407317702494 us; error 36 us; skew 500 ppm
Sep 26 07:21:57 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407317, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:21:57.705667 9346 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:21:57.706673 9346 webserver.cc:460] Webserver started at http://127.28.235.254:33331/ using document root <none> and password file <none>
I20260926 07:21:57.707249 9346 fs_manager.cc:359] Metadata directory not provided
I20260926 07:21:57.707340 9346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:21:57.707583 9346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:21:57 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407317, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:21:57.758986 9346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "debb0d8c02514b878568a74f3c853f30"
format_stamp: "Formatted at 2026-09-26 07:21:57 on dist-test-slave-7zlw"
server_key: "3fc0776bb4d37cff356ec46da1a75be0"
server_key_iv: "6729ddedaf52b8fff3277fbb9f9e3fc1"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:57.759654 9346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "debb0d8c02514b878568a74f3c853f30"
format_stamp: "Formatted at 2026-09-26 07:21:57 on dist-test-slave-7zlw"
server_key: "3fc0776bb4d37cff356ec46da1a75be0"
server_key_iv: "6729ddedaf52b8fff3277fbb9f9e3fc1"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:57.764017 9346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 07:21:57.779269 9361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:57.780930 9346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:21:57.781139 9346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "debb0d8c02514b878568a74f3c853f30"
format_stamp: "Formatted at 2026-09-26 07:21:57 on dist-test-slave-7zlw"
server_key: "3fc0776bb4d37cff356ec46da1a75be0"
server_key_iv: "6729ddedaf52b8fff3277fbb9f9e3fc1"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:57.781308 9346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:21:57.811872 9346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:21:57.815327 9346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:21:57.815567 9346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:21:57.824021 9346 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:37009
I20260926 07:21:57.824024 9413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:37009 every 8 connection(s)
I20260926 07:21:57.825217 9346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:21:57.828167 9414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:21:57.832948 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 9346
I20260926 07:21:57.833092 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:21:57.833384 29615 external_mini_cluster.cc:1428] Setting key 15ea5d419ef956d51f44ee478b8d71ca
I20260926 07:21:57.834836 9414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: Bootstrap starting.
I20260926 07:21:57.837386 9414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: Neither blocks nor log segments found. Creating new log.
I20260926 07:21:57.838415 9414 log.cc:824] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: Log is configured to *not* fsync() on all Append() calls
I20260926 07:21:57.840741 9414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: No bootstrap required, opened a new log
Sep 26 07:21:57 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:21:57.845501 9414 raft_consensus.cc:348] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } }
I20260926 07:21:57.845845 9414 raft_consensus.cc:374] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:21:57.845916 9414 raft_consensus.cc:729] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debb0d8c02514b878568a74f3c853f30, State: Initialized, Role: FOLLOWER
I20260926 07:21:57.846565 9414 consensus_queue.cc:260] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } }
I20260926 07:21:57.846748 9414 raft_consensus.cc:388] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:21:57.846854 9414 raft_consensus.cc:482] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:21:57.847005 9414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:21:57.848201 9414 raft_consensus.cc:504] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } }
I20260926 07:21:57.848652 9414 leader_election.cc:302] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: debb0d8c02514b878568a74f3c853f30; no voters:
I20260926 07:21:57.849005 9414 leader_election.cc:288] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:21:57.849203 9419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:21:57.849476 9419 raft_consensus.cc:686] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 1 LEADER]: Becoming Leader. State: Replica: debb0d8c02514b878568a74f3c853f30, State: Running, Role: LEADER
I20260926 07:21:57.849856 9419 consensus_queue.cc:237] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } }
I20260926 07:21:57.850565 9414 sys_catalog.cc:564] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:21:57.851265 9419 sys_catalog.cc:455] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "debb0d8c02514b878568a74f3c853f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } } }
I20260926 07:21:57.851455 9419 sys_catalog.cc:458] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [sys.catalog]: This master's current role is: LEADER
I20260926 07:21:57.851590 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:57.834964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.28.235.254:37009)
0926 07:21:57.835473 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:57.835493 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:57.835540 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:21:57.836414 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:57.836441 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:57.836903 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:57.837324 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:21:57.838288 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:57.839776 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:57.841040 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:57.841751 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:57.841916 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:57.845184 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:21:57.845191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:21:57.845209 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:21:57.845244 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:21:57.847442 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:57.847988 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:57.847996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:57.848002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:57.848149 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:57.848472 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:57.848476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:57.848477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:57.849108 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:21:57.850092 (+ 984us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:57.850252 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 07:21:57.850549 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":149,"threads_started":1}
I20260926 07:21:57.851683 9419 sys_catalog.cc:455] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debb0d8c02514b878568a74f3c853f30. Latest consensus state: current_term: 1 leader_uuid: "debb0d8c02514b878568a74f3c853f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } } }
I20260926 07:21:57.852135 9419 sys_catalog.cc:458] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [sys.catalog]: This master's current role is: LEADER
I20260926 07:21:57.851951 9427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:21:57.855562 9427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:21:57.856722 9414 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:21:57.864966 9427 catalog_manager.cc:1347] Generated new cluster ID: f6f1ea5958b749479f948e9c0afb3d68
I20260926 07:21:57.865188 9427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:21:57.879733 9427 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:21:57.881161 9427 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:21:57.894726 9427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: Generated new TSK 0
I20260926 07:21:57.895697 9427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:21:58.737901 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:45715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:37009
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:21:58.889781 9459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:21:58.890137 9459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:21:58.890231 9459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:21:58.895676 9459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:21:58.895829 9459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:21:58.895947 9459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:21:58.904527 9459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.235.212:45715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:37009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:21:58.906170 9459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:21:58.907855 9459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:21:58.921916 9465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:58.922118 9464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:58.922474 9467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:21:58.922956 9459 server_base.cc:1034] running on GCE node
I20260926 07:21:58.923427 9459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:21:58.924211 9459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:21:58.925406 9459 hybrid_clock.cc:648] HybridClock initialized: now 1790407318925374 us; error 46 us; skew 500 ppm
Sep 26 07:21:58 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407318, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:21:58.929355 9459 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:21:58.931137 9459 webserver.cc:460] Webserver started at http://127.28.235.193:46399/ using document root <none> and password file <none>
I20260926 07:21:58.932044 9459 fs_manager.cc:359] Metadata directory not provided
I20260926 07:21:58.932164 9459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:21:58.932531 9459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:21:58 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407318, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:21:58.954000 9459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "f81858e64f984b3cb2fcc559e4d8b9ba"
format_stamp: "Formatted at 2026-09-26 07:21:58 on dist-test-slave-7zlw"
server_key: "3c5f4b6dbd7c929e059fc5b95aa37478"
server_key_iv: "f1b085ff7fd13d108f4481114955050c"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:58.955003 9459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "f81858e64f984b3cb2fcc559e4d8b9ba"
format_stamp: "Formatted at 2026-09-26 07:21:58 on dist-test-slave-7zlw"
server_key: "3c5f4b6dbd7c929e059fc5b95aa37478"
server_key_iv: "f1b085ff7fd13d108f4481114955050c"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:58.960568 9459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 07:21:58.988162 9476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:58.990048 9459 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 07:21:58.990198 9459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "f81858e64f984b3cb2fcc559e4d8b9ba"
format_stamp: "Formatted at 2026-09-26 07:21:58 on dist-test-slave-7zlw"
server_key: "3c5f4b6dbd7c929e059fc5b95aa37478"
server_key_iv: "f1b085ff7fd13d108f4481114955050c"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:58.990339 9459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:21:59.019235 9459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:21:59.023367 9459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:21:59.023836 9459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:21:59.029937 9459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:21:59.031209 9459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:21:59.037099 9459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 07:21:59.037317 9459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:21:59.037415 9459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:59.048339 9459 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:34065
I20260926 07:21:59.049772 9459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:21:59.052237 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 9459
I20260926 07:21:59.052368 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:21:59.052717 29615 external_mini_cluster.cc:1428] Setting key 167561479756b8b42fb5ef9370895e52
I20260926 07:21:59.058271 9589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:34065 every 8 connection(s)
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
Sep 26 07:21:59 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407318, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:21:59.091944 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:59.066592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:34841 (local address 127.28.235.254:37009)
0926 07:21:59.072689 (+ 6097us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:59.072694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:59.075101 (+ 2407us) server_negotiation.cc:407] Connection header received
0926 07:21:59.075270 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:59.075276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:59.075349 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:59.075446 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:21:59.077281 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.078163 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.079936 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.080290 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.080378 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:59.087090 (+ 6712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:21:59.087095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:21:59.087098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:21:59.087131 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:21:59.089190 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:59.089923 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:59.089927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:59.089929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:59.090023 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:59.090478 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:59.090482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:59.090484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:59.090724 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:21:59.090825 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:59.091590 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 07:21:59.091733 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5963,"thread_start_us":97,"threads_started":1}
I20260926 07:21:59.094009 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:59.065213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:37009 (local address 127.28.235.193:34841)
0926 07:21:59.073314 (+ 8101us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:59.073413 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:59.074710 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:59.075825 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:59.075838 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:59.076248 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:21:59.077084 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:59.077112 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.078335 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.078338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:59.079762 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:59.079774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.080619 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.080624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:59.080835 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:59.084801 (+ 3966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:21:59.084829 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:21:59.086824 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:21:59.089359 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:59.089395 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:59.089415 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:59.089793 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:59.090132 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:59.090134 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:59.090136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:59.090350 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:59.090895 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:21:59.090902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:21:59.091270 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 07:21:59.092448 (+ 1178us) client_negotiation.cc:240] Negotiation successful
0926 07:21:59.092844 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7664,"thread_start_us":171,"threads_started":1}
I20260926 07:21:59.095863 9590 heartbeater.cc:347] Connected to a master server at 127.28.235.254:37009
I20260926 07:21:59.096124 9590 heartbeater.cc:464] Registering TS with master...
I20260926 07:21:59.096728 9590 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
I20260926 07:21:59.098675 9378 ts_manager.cc:194] Registered new tserver with Master: f81858e64f984b3cb2fcc559e4d8b9ba (127.28.235.193:34065)
I20260926 07:21:59.100256 9378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:34841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:21:59.122268 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:45715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:37009
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:21:59 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 07:21:59.293511 9598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:21:59.293825 9598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:21:59.293946 9598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:21:59.299069 9598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:21:59.299223 9598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:21:59.299340 9598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:21:59.307662 9598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.235.212:45715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:37009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:21:59.309423 9598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:21:59.310979 9598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:21:59.325297 9605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:59.325845 9604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:59.326408 9607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:21:59.326925 9598 server_base.cc:1034] running on GCE node
I20260926 07:21:59.327492 9598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:21:59.328328 9598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:21:59.329617 9598 hybrid_clock.cc:648] HybridClock initialized: now 1790407319329548 us; error 87 us; skew 500 ppm
Sep 26 07:21:59 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:21:59.333627 9598 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:21:59.334774 9598 webserver.cc:460] Webserver started at http://127.28.235.194:36121/ using document root <none> and password file <none>
I20260926 07:21:59.335530 9598 fs_manager.cc:359] Metadata directory not provided
I20260926 07:21:59.335608 9598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:21:59.335857 9598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:21:59 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:21:59.358513 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "7b7d50af2e384d9e965649cc9c85430d"
format_stamp: "Formatted at 2026-09-26 07:21:59 on dist-test-slave-7zlw"
server_key: "9174d84d43a93e754f90aa8ce0e59806"
server_key_iv: "d3b86e89cafdc2716fe57a726372e0d9"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:59.359692 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "7b7d50af2e384d9e965649cc9c85430d"
format_stamp: "Formatted at 2026-09-26 07:21:59 on dist-test-slave-7zlw"
server_key: "9174d84d43a93e754f90aa8ce0e59806"
server_key_iv: "d3b86e89cafdc2716fe57a726372e0d9"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:59.365978 9598 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 07:21:59.391556 9614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:59.393446 9598 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:21:59.393589 9598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "7b7d50af2e384d9e965649cc9c85430d"
format_stamp: "Formatted at 2026-09-26 07:21:59 on dist-test-slave-7zlw"
server_key: "9174d84d43a93e754f90aa8ce0e59806"
server_key_iv: "d3b86e89cafdc2716fe57a726372e0d9"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:59.393729 9598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:21:59.426908 9598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:21:59.430882 9598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:21:59.431147 9598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:21:59.431918 9598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:21:59.433092 9598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:21:59.433207 9598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:59.433310 9598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:21:59.433383 9598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:59.448035 9598 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:45177
I20260926 07:21:59.449510 9598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:21:59.449577 9727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:45177 every 8 connection(s)
I20260926 07:21:59.456933 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 9598
I20260926 07:21:59.457183 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:21:59.457531 29615 external_mini_cluster.cc:1428] Setting key bb5ef2676983145f65ba80a6cacfb22c
Sep 26 07:21:59 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:21:59.475809 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:59.453041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:36189 (local address 127.28.235.254:37009)
0926 07:21:59.453180 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:59.453185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:59.454927 (+ 1742us) server_negotiation.cc:407] Connection header received
0926 07:21:59.456243 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:59.456248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:59.456309 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:59.456364 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:21:59.462863 (+ 6499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.463726 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.465562 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.465951 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.466034 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:59.469348 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:21:59.469352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:21:59.469355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:21:59.469417 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:21:59.471242 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:59.473128 (+ 1886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:59.473132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:59.473134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:59.473233 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:59.473689 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:59.473693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:59.473695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:59.473951 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:21:59.474038 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:59.475464 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0926 07:21:59.475626 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:21:59.477928 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:59.453449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:37009 (local address 127.28.235.194:36189)
0926 07:21:59.454728 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:59.454790 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:59.455967 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:59.461471 (+ 5504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:59.461483 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:59.461891 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:21:59.462645 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:59.462665 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.463916 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.463920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:59.465415 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:59.465426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.466257 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.466261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:59.466471 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:59.467232 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:21:59.467254 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:21:59.469131 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:21:59.471400 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:59.471407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:59.471425 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:59.471785 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:59.473354 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:59.473357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:59.473359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:59.473572 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:59.474110 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:21:59.474118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:21:59.474471 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 07:21:59.476320 (+ 1849us) client_negotiation.cc:240] Negotiation successful
0926 07:21:59.476711 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"thread_start_us":156,"threads_started":1}
I20260926 07:21:59.480506 9728 heartbeater.cc:347] Connected to a master server at 127.28.235.254:37009
I20260926 07:21:59.480804 9728 heartbeater.cc:464] Registering TS with master...
I20260926 07:21:59.481534 9728 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
I20260926 07:21:59.482883 9378 ts_manager.cc:194] Registered new tserver with Master: 7b7d50af2e384d9e965649cc9c85430d (127.28.235.194:45177)
I20260926 07:21:59.483630 9378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:36189
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:21:59.509648 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:45715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:37009
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:21:59.682592 9733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:21:59.683130 9733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:21:59.683283 9733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:21:59.690155 9733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:21:59.690310 9733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:21:59.690459 9733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:21:59.699059 9733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.235.212:45715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:37009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:21:59.700919 9733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:21:59.702159 9733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:21:59.715823 9739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:21:59.716656 9733 server_base.cc:1034] running on GCE node
W20260926 07:21:59.717247 9741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:21:59.717510 9738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:21:59.718179 9733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:21:59.719082 9733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:21:59.735252 9733 hybrid_clock.cc:648] HybridClock initialized: now 1790407319735146 us; error 107 us; skew 500 ppm
Sep 26 07:21:59 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:21:59.739488 9733 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:21:59.744707 9733 webserver.cc:460] Webserver started at http://127.28.235.195:38375/ using document root <none> and password file <none>
I20260926 07:21:59.745651 9733 fs_manager.cc:359] Metadata directory not provided
I20260926 07:21:59.745781 9733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:21:59.746069 9733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:21:59 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
Sep 26 07:21:59 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:21:59.765954 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "310f94fb6e1140ffaf8c63134b031d80"
format_stamp: "Formatted at 2026-09-26 07:21:59 on dist-test-slave-7zlw"
server_key: "3e8cf873edc2ee47486c2e08eea682ff"
server_key_iv: "67c282951572159b6760916da3fc5740"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:59.766865 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "310f94fb6e1140ffaf8c63134b031d80"
format_stamp: "Formatted at 2026-09-26 07:21:59 on dist-test-slave-7zlw"
server_key: "3e8cf873edc2ee47486c2e08eea682ff"
server_key_iv: "67c282951572159b6760916da3fc5740"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:59.773736 9733 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 07:21:59.793838 9748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:59.795887 9733 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260926 07:21:59.796118 9733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "310f94fb6e1140ffaf8c63134b031d80"
format_stamp: "Formatted at 2026-09-26 07:21:59 on dist-test-slave-7zlw"
server_key: "3e8cf873edc2ee47486c2e08eea682ff"
server_key_iv: "67c282951572159b6760916da3fc5740"
server_key_version: "kuduclusterkey@0"
I20260926 07:21:59.796380 9733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:21:59.844173 9733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:21:59.848481 9733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:21:59.848767 9733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:21:59.849596 9733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:21:59.850804 9733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:21:59.850914 9733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:59.851014 9733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:21:59.851074 9733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:21:59.861475 9733 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:40451
I20260926 07:21:59.862766 9733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:21:59.870646 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 9733
I20260926 07:21:59.870805 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:21:59.871107 29615 external_mini_cluster.cc:1428] Setting key 14a6d259c7e8c46d62460422c48ca8d5
I20260926 07:21:59.877341 9861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:40451 every 8 connection(s)
Sep 26 07:21:59 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:21:59.898972 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:59.881292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:43415 (local address 127.28.235.254:37009)
0926 07:21:59.881455 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 07:21:59.881460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:21:59.882570 (+ 1110us) server_negotiation.cc:407] Connection header received
0926 07:21:59.886045 (+ 3475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:21:59.886049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:21:59.886107 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:21:59.886157 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:21:59.887995 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.888772 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.890537 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.890864 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.890978 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:59.894172 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:21:59.894177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:21:59.894179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:21:59.894212 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:21:59.895957 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:59.896776 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:59.896780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:59.896781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:59.896870 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:21:59.897352 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:21:59.897356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:21:59.897357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:21:59.897587 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:21:59.897668 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:21:59.898674 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0926 07:21:59.898812 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:21:59.901332 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:21:59.880495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:37009 (local address 127.28.235.195:43415)
0926 07:21:59.882382 (+ 1887us) negotiation.cc:107] Waiting for socket to connect
0926 07:21:59.882448 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 07:21:59.885771 (+ 3323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:21:59.886633 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:21:59.886644 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:21:59.887034 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:21:59.887777 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:59.887797 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.888940 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.888944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:59.890365 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:21:59.890377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:21:59.891049 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:21:59.891053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:21:59.891257 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:21:59.892046 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:21:59.892069 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:21:59.893943 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:21:59.896152 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:59.896159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:59.896179 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:59.896654 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:59.896982 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:21:59.896985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:21:59.896987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:21:59.897229 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:21:59.897948 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:21:59.897955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:21:59.898349 (+ 394us) client_negotiation.cc:769] Sending connection context
0926 07:21:59.899604 (+ 1255us) client_negotiation.cc:240] Negotiation successful
0926 07:21:59.900018 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1631,"thread_start_us":189,"threads_started":1}
I20260926 07:21:59.903067 9862 heartbeater.cc:347] Connected to a master server at 127.28.235.254:37009
I20260926 07:21:59.903365 9862 heartbeater.cc:464] Registering TS with master...
I20260926 07:21:59.904019 9862 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
I20260926 07:21:59.905377 9378 ts_manager.cc:194] Registered new tserver with Master: 310f94fb6e1140ffaf8c63134b031d80 (127.28.235.195:40451)
I20260926 07:21:59.906201 9378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:43415
I20260926 07:21:59.909428 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:21:59.910308 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 9459
I20260926 07:21:59.918341 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 9598
I20260926 07:21:59.926396 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 9733
I20260926 07:21:59.934376 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 9346
I20260926 07:21:59.942149 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:37009
--webserver_interface=127.28.235.254
--webserver_port=33331
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:37009
--ranger_config_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:45715/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.235.212:45715/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:00.066425 9865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:00.066727 9865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:00.066768 9865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:00.071179 9865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:22:00.071259 9865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:00.071277 9865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:00.071295 9865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:22:00.071309 9865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:22:00.077287 9865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.235.212:45715/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:37009
--ranger_config_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:37009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=33331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:00.078500 9865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:00.079675 9865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:00.089907 9870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:00.089974 9871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:00.089910 9873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:00.090691 9865 server_base.cc:1034] running on GCE node
I20260926 07:22:00.091372 9865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:00.092628 9865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:00.093948 9865 hybrid_clock.cc:648] HybridClock initialized: now 1790407320093927 us; error 42 us; skew 500 ppm
Sep 26 07:22:00 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:00.097218 9865 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:22:00.098388 9865 webserver.cc:460] Webserver started at http://127.28.235.254:33331/ using document root <none> and password file <none>
I20260926 07:22:00.099053 9865 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:00.099164 9865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:22:00 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:22:00.147672 9865 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:00.150205 9881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:00.151674 9865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:00.151849 9865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "debb0d8c02514b878568a74f3c853f30"
format_stamp: "Formatted at 2026-09-26 07:21:57 on dist-test-slave-7zlw"
server_key: "3fc0776bb4d37cff356ec46da1a75be0"
server_key_iv: "6729ddedaf52b8fff3277fbb9f9e3fc1"
server_key_version: "kuduclusterkey@0"
I20260926 07:22:00.152429 9865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:00.170864 9865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:00.174453 9865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:00.174677 9865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:00.182739 9865 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:37009
I20260926 07:22:00.182786 9933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:37009 every 8 connection(s)
I20260926 07:22:00.183871 9865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:22:00.188417 9934 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:22:00.190243 9934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: Bootstrap starting.
I20260926 07:22:00.191728 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 9865
I20260926 07:22:00.192272 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:45715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:34065
--local_ip_for_outbound_sockets=127.28.235.193
--tserver_master_addrs=127.28.235.254:37009
--webserver_port=46399
--webserver_interface=127.28.235.193
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.235.212:45715/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:22:00.207513 9934 log.cc:824] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:00.214677 9934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:22:00.215140 9934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: Bootstrap complete.
I20260926 07:22:00.219494 9934 raft_consensus.cc:348] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } }
I20260926 07:22:00.219954 9934 raft_consensus.cc:729] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: debb0d8c02514b878568a74f3c853f30, State: Initialized, Role: FOLLOWER
I20260926 07:22:00.220475 9934 consensus_queue.cc:260] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } }
I20260926 07:22:00.220629 9934 raft_consensus.cc:388] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:22:00.220723 9934 raft_consensus.cc:482] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:22:00.220846 9934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:22:00.222921 9934 raft_consensus.cc:504] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } }
I20260926 07:22:00.223287 9934 leader_election.cc:302] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: debb0d8c02514b878568a74f3c853f30; no voters:
I20260926 07:22:00.223658 9934 leader_election.cc:288] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:22:00.223757 9938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:22:00.223968 9938 raft_consensus.cc:686] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [term 2 LEADER]: Becoming Leader. State: Replica: debb0d8c02514b878568a74f3c853f30, State: Running, Role: LEADER
I20260926 07:22:00.224263 9938 consensus_queue.cc:237] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } }
I20260926 07:22:00.225092 9934 sys_catalog.cc:564] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:22:00.225726 9939 sys_catalog.cc:455] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "debb0d8c02514b878568a74f3c853f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } } }
I20260926 07:22:00.225862 9939 sys_catalog.cc:458] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:00.226485 9940 sys_catalog.cc:455] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debb0d8c02514b878568a74f3c853f30. Latest consensus state: current_term: 2 leader_uuid: "debb0d8c02514b878568a74f3c853f30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debb0d8c02514b878568a74f3c853f30" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 37009 } } }
I20260926 07:22:00.226580 9940 sys_catalog.cc:458] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:00.226737 9945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:22:00.230760 9945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:22:00.231309 9934 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:22:00.231632 9945 catalog_manager.cc:1259] Loaded cluster ID: f6f1ea5958b749479f948e9c0afb3d68
I20260926 07:22:00.231751 9945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:22:00.234194 9945 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:22:00.234918 9945 catalog_manager.cc:5879] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: Loaded TSK: 0
I20260926 07:22:00.236191 9945 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:22:00.340344 9936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:00.340772 9936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:00.340842 9936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:00.348165 9936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:00.348268 9936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:00.348363 9936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:22:00.357087 9936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.235.212:45715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:34065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=46399
--tserver_master_addrs=127.28.235.254:37009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:00.358592 9936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:00.360060 9936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:00.373896 9972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:00.373955 9971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:00.374398 9936 server_base.cc:1034] running on GCE node
W20260926 07:22:00.374404 9974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:00.375137 9936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:00.376073 9936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:00.377290 9936 hybrid_clock.cc:648] HybridClock initialized: now 1790407320377213 us; error 91 us; skew 500 ppm
Sep 26 07:22:00 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:00.381304 9936 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:22:00.382755 9936 webserver.cc:460] Webserver started at http://127.28.235.193:46399/ using document root <none> and password file <none>
I20260926 07:22:00.383540 9936 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:00.383647 9936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:22:00 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:22:00.427618 9936 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.002s sys 0.000s
I20260926 07:22:00.431409 9982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:00.433327 9936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 07:22:00.433480 9936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "f81858e64f984b3cb2fcc559e4d8b9ba"
format_stamp: "Formatted at 2026-09-26 07:21:58 on dist-test-slave-7zlw"
server_key: "3c5f4b6dbd7c929e059fc5b95aa37478"
server_key_iv: "f1b085ff7fd13d108f4481114955050c"
server_key_version: "kuduclusterkey@0"
I20260926 07:22:00.434295 9936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:00.460564 9936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:00.464483 9936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:00.464730 9936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:00.465627 9936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:00.466833 9936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:00.466909 9936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:00.466985 9936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:00.467023 9936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:00.489081 9936 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:34065
I20260926 07:22:00.489194 10095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:34065 every 8 connection(s)
I20260926 07:22:00.490533 9936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:22:00.495553 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 9936
I20260926 07:22:00.496165 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:45715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:45177
--local_ip_for_outbound_sockets=127.28.235.194
--tserver_master_addrs=127.28.235.254:37009
--webserver_port=36121
--webserver_interface=127.28.235.194
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.235.212:45715/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:22:00 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:00.521370 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:00.493339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:56267 (local address 127.28.235.254:37009)
0926 07:22:00.494022 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:00.494048 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:00.494571 (+ 523us) server_negotiation.cc:407] Connection header received
0926 07:22:00.498222 (+ 3651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:00.498260 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:00.498817 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:00.499199 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:00.502271 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:00.503976 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:00.505837 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:00.506623 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:00.506775 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:00.514318 (+ 7543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:00.514328 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:00.514353 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:00.514400 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:00.516781 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:00.517599 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:00.517606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:00.517613 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:00.517773 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:00.518220 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:00.518223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:00.518225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:00.518787 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:00.518918 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:00.519755 (+ 837us) server_negotiation.cc:299] Negotiation successful
0926 07:22:00.520130 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":122,"threads_started":1}
I20260926 07:22:00.523382 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:00.493544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:37009 (local address 127.28.235.193:56267)
0926 07:22:00.494366 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:00.494428 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:00.496324 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:00.500832 (+ 4508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:00.500842 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:00.501282 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:00.502016 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:00.502038 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:00.504173 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:00.504176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:00.505674 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:00.505687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:00.507073 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:00.507077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:00.507294 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:00.511990 (+ 4696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:00.512019 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:00.513947 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:00.516969 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:00.516978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:00.516996 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:00.517440 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:00.517909 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:00.517912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:00.517914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:00.518104 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:00.518993 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:00.519000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:00.519354 (+ 354us) client_negotiation.cc:769] Sending connection context
0926 07:22:00.521937 (+ 2583us) client_negotiation.cc:240] Negotiation successful
0926 07:22:00.522353 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":170,"threads_started":1}
I20260926 07:22:00.526739 10096 heartbeater.cc:347] Connected to a master server at 127.28.235.254:37009
I20260926 07:22:00.527040 10096 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:00.527700 10096 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
W20260926 07:22:00.530390 10096 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:37009 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:22:00.530542 10096 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:00.530812 10096 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
I20260926 07:22:00.531795 10096 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:00.532001 10096 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
W20260926 07:22:00.532873 10096 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:22:00.649345 10100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:00.649785 10100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:00.649883 10100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:00.654556 10100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:00.654651 10100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:00.654754 10100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:22:00.660996 10100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.235.212:45715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:45177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=36121
--tserver_master_addrs=127.28.235.254:37009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:00.662468 10100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:00.663865 10100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:00.676520 10106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:00.676614 10105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:00.677065 10100 server_base.cc:1034] running on GCE node
W20260926 07:22:00.683368 10108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:00.684118 10100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:00.685401 10100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:00.686632 10100 hybrid_clock.cc:648] HybridClock initialized: now 1790407320686560 us; error 93 us; skew 500 ppm
Sep 26 07:22:00 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:00.690706 10100 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:22:00.692070 10100 webserver.cc:460] Webserver started at http://127.28.235.194:36121/ using document root <none> and password file <none>
I20260926 07:22:00.692871 10100 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:00.692992 10100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:22:00 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:22:00.727880 10100 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.000s
I20260926 07:22:00.734524 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:00.736267 10100 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260926 07:22:00.736497 10100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "7b7d50af2e384d9e965649cc9c85430d"
format_stamp: "Formatted at 2026-09-26 07:21:59 on dist-test-slave-7zlw"
server_key: "9174d84d43a93e754f90aa8ce0e59806"
server_key_iv: "d3b86e89cafdc2716fe57a726372e0d9"
server_key_version: "kuduclusterkey@0"
I20260926 07:22:00.737403 10100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:00.766980 10100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:00.770860 10100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:00.771134 10100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:00.771955 10100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:00.773219 10100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:00.773335 10100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:00.773434 10100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:00.773497 10100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:00.787410 10100 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:45177
I20260926 07:22:00.788635 10100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:22:00.787425 10230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:45177 every 8 connection(s)
I20260926 07:22:00.793128 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 10100
I20260926 07:22:00.793659 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.235.212:45715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:40451
--local_ip_for_outbound_sockets=127.28.235.195
--tserver_master_addrs=127.28.235.254:37009
--webserver_port=38375
--webserver_interface=127.28.235.195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:32899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.235.212:45715/kms with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:22:00 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407320, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:00.808017 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:00.790854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:48467 (local address 127.28.235.254:37009)
0926 07:22:00.791000 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:00.791005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:00.792053 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 07:22:00.793399 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:00.793405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:00.793475 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:00.793532 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:00.795716 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:00.796614 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:00.797932 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:00.798355 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:00.798476 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:00.801636 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:00.801641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:00.801644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:00.801684 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:00.803631 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:00.804455 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:00.804460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:00.804463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:00.804576 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:00.805125 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:00.805130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:00.805134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:00.805395 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:00.805489 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:00.806273 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 07:22:00.806600 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:22:00.810680 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:00.791363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:37009 (local address 127.28.235.194:48467)
0926 07:22:00.791865 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:00.791928 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:00.793049 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:00.794263 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:00.794276 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:00.794647 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:00.795502 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:00.795528 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:00.796804 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:00.796807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:00.797780 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:00.797798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:00.798685 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:00.798691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:00.798933 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:00.799696 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:00.799721 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:00.801408 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:00.803820 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:00.803831 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:00.803858 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:00.804312 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:00.804710 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:00.804715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:00.804718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:00.804950 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:00.805563 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:00.805570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:00.805959 (+ 389us) client_negotiation.cc:769] Sending connection context
0926 07:22:00.808939 (+ 2980us) client_negotiation.cc:240] Negotiation successful
0926 07:22:00.809462 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":152,"threads_started":1}
I20260926 07:22:00.812640 10231 heartbeater.cc:347] Connected to a master server at 127.28.235.254:37009
I20260926 07:22:00.812935 10231 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:00.813757 10231 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
W20260926 07:22:00.815862 10231 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:37009 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:22:00.816054 10231 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:00.816296 10231 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
I20260926 07:22:00.817312 10231 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:00.817526 10231 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
W20260926 07:22:00.818406 10231 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:22:00.980192 10234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:00.980623 10234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:00.980701 10234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:00.987408 10234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:00.987534 10234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:00.987636 10234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:22:00.996275 10234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:32899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.235.212:45715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:40451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=38375
--tserver_master_addrs=127.28.235.254:37009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:00.997872 10234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:00.999311 10234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:01.012943 10242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:01.013825 10239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:01.013883 10234 server_base.cc:1034] running on GCE node
W20260926 07:22:01.017325 10240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:01.018082 10234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:01.019058 10234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:01.020284 10234 hybrid_clock.cc:648] HybridClock initialized: now 1790407321020241 us; error 62 us; skew 500 ppm
Sep 26 07:22:01 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407321, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:01.023741 10234 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:22:01.024945 10234 webserver.cc:460] Webserver started at http://127.28.235.195:38375/ using document root <none> and password file <none>
I20260926 07:22:01.025748 10234 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:01.025887 10234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:22:01 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407321, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:22:01.050438 10234 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:22:01.053793 10250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:01.055408 10234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:22:01.055601 10234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "310f94fb6e1140ffaf8c63134b031d80"
format_stamp: "Formatted at 2026-09-26 07:21:59 on dist-test-slave-7zlw"
server_key: "3e8cf873edc2ee47486c2e08eea682ff"
server_key_iv: "67c282951572159b6760916da3fc5740"
server_key_version: "kuduclusterkey@0"
I20260926 07:22:01.056349 10234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:01.078985 10234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:01.082533 10234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:01.082773 10234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:01.083560 10234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:01.084540 10234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:01.084616 10234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:01.084717 10234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:01.084770 10234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:01.096726 10234 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:40451
I20260926 07:22:01.096809 10363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:40451 every 8 connection(s)
I20260926 07:22:01.097999 10234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:22:01.107759 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 10234
Sep 26 07:22:01 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407321, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:01.116510 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:01.100023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:45313 (local address 127.28.235.254:37009)
0926 07:22:01.100159 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:01.100163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:01.102058 (+ 1895us) server_negotiation.cc:407] Connection header received
0926 07:22:01.103354 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:01.103358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:01.103420 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:01.103482 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:01.105073 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.105956 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.107689 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.108015 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.108138 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:01.112095 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:01.112099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:01.112102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:01.112129 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:01.113935 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:01.114639 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:01.114643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:01.114645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:01.114735 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:01.115183 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:01.115187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:01.115189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:01.115477 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:01.115585 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:01.116268 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 07:22:01.116384 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:22:01.117772 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:01.100490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:37009 (local address 127.28.235.195:45313)
0926 07:22:01.101866 (+ 1376us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:01.101930 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:01.103070 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:01.103859 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:01.103868 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:01.104226 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:01.104853 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:01.104873 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.106113 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.106117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:01.107532 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:01.107544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.108549 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.108557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:01.108767 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:01.109939 (+ 1172us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:01.109964 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:01.111873 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:01.114056 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:01.114064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:01.114084 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:01.114507 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:01.114858 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:01.114863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:01.114867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:01.115076 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:01.115610 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:01.115618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:01.115991 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 07:22:01.116219 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 07:22:01.116616 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"thread_start_us":165,"threads_started":1}
I20260926 07:22:01.119323 10364 heartbeater.cc:347] Connected to a master server at 127.28.235.254:37009
I20260926 07:22:01.119593 10364 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:01.120221 10364 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
I20260926 07:22:01.121840 9898 ts_manager.cc:194] Registered new tserver with Master: 310f94fb6e1140ffaf8c63134b031d80 (127.28.235.195:40451)
I20260926 07:22:01.123421 9898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:45313
I20260926 07:22:01.127046 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:01.108551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.28.235.254:37009)
0926 07:22:01.109756 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:01.109767 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:01.109783 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:22:01.109885 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:01.109888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:01.109941 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:01.110035 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:01.111087 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.111882 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.112972 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.113378 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.113508 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:01.115099 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:01.115104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:01.115106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:01.115132 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:01.125190 (+ 10058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:01.125706 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:01.125710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:01.125712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:01.125805 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:01.126106 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:01.126111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:01.126116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:01.126306 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:01.126507 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:01.126723 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 07:22:01.126854 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":111,"threads_started":1}
Sep 26 07:22:01 dist-test-slave-7zlw krb5kdc[8902](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407321, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:01.533159 10096 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:01.533464 10096 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
I20260926 07:22:01.534191 9898 ts_manager.cc:194] Registered new tserver with Master: f81858e64f984b3cb2fcc559e4d8b9ba (127.28.235.193:34065)
I20260926 07:22:01.535004 9898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:56267
I20260926 07:22:01.818687 10231 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:01.819029 10231 heartbeater.cc:511] Master 127.28.235.254:37009 requested a full tablet report, sending...
I20260926 07:22:01.820073 9898 ts_manager.cc:194] Registered new tserver with Master: 7b7d50af2e384d9e965649cc9c85430d (127.28.235.194:45177)
I20260926 07:22:01.820896 9898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:48467
I20260926 07:22:01.826501 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:22:01.840209 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:01.828560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.28.235.254:37009)
0926 07:22:01.829228 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:01.829233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:01.829651 (+ 418us) server_negotiation.cc:407] Connection header received
0926 07:22:01.829700 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:01.829703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:01.829759 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:01.829871 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:01.830868 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.831697 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.833000 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.833382 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.833501 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:01.835306 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:01.835311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:01.835314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:01.835346 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:01.837558 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:01.838428 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:01.838433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:01.838435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:01.838541 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:01.838937 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:01.838941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:01.838943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:01.839213 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:01.839359 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:01.839662 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 07:22:01.839828 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":79,"threads_started":1}
I20260926 07:22:01.849988 9898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:22:01.855575 9898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:22:01.879000 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:01.869845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:40451 (local address 127.0.0.1:38846)
0926 07:22:01.871273 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:01.871288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:01.871384 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:01.872507 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:01.872515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:01.872540 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:01.872717 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:01.872725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.875217 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.875220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:01.877175 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:01.877185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.878473 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.878477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:01.878574 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:01.878599 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:22:01.878675 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 07:22:01.878833 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"mutex_wait_us":814,"thread_start_us":97,"threads_started":1}
I20260926 07:22:01.879057 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:01.870590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.28.235.194:45177)
0926 07:22:01.871478 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:01.871486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:01.872682 (+ 1196us) server_negotiation.cc:407] Connection header received
0926 07:22:01.872771 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:01.872779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:01.872998 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:01.873162 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:01.873597 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.874606 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.876171 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.877047 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.877172 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:01.877339 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:01.878794 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0926 07:22:01.878920 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":121,"threads_started":1}
I20260926 07:22:01.879765 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:01.870138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:45177 (local address 127.0.0.1:59372)
0926 07:22:01.872074 (+ 1936us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:01.872089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:01.872190 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:01.873335 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:01.873338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:01.873356 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:01.873463 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:01.873468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.874729 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.874733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:01.876047 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:01.876059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.877401 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.877404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:01.877513 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:01.877553 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 07:22:01.878666 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0926 07:22:01.879621 (+ 955us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1863,"mutex_wait_us":847,"thread_start_us":130,"threads_started":1}
I20260926 07:22:01.880997 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:01.870979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38846 (local address 127.28.235.195:40451)
0926 07:22:01.871676 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:01.871682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:01.871708 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:22:01.871795 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:01.871801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:01.872271 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:01.872522 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:01.872870 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.874915 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.877322 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.878146 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.878264 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:01.878411 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:01.880741 (+ 2330us) server_negotiation.cc:299] Negotiation successful
0926 07:22:01.880860 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":125,"threads_started":1}
I20260926 07:22:01.883212 10164 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3e8eb4b9df4181803b688f9bde60e3 (DEFAULT_TABLE table=test-table [id=061ba2b6504a4579b2a7935fa29927e0]), partition=RANGE (key) PARTITION UNBOUNDED
Sep 26 07:22:01 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407321, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:22:01.883884 10298 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3e8eb4b9df4181803b688f9bde60e3 (DEFAULT_TABLE table=test-table [id=061ba2b6504a4579b2a7935fa29927e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:01.884618 10164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3e8eb4b9df4181803b688f9bde60e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:01.884932 10298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3e8eb4b9df4181803b688f9bde60e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:01.894797 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:01.868362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:34065 (local address 127.0.0.1:37450)
0926 07:22:01.868779 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:01.868827 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:01.868949 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:01.886011 (+ 17062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:01.886015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:01.886045 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:01.886169 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:01.886181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.890716 (+ 4535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.890721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:01.891916 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:01.891934 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.894311 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.894316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:01.894435 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:01.894468 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:22:01.894526 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:22:01.894623 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":142,"threads_started":1}
I20260926 07:22:01.895797 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:01.869265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.28.235.193:34065)
0926 07:22:01.885471 (+ 16206us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:01.885479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:01.885512 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:22:01.885604 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:01.885610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:01.885830 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:01.886019 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:01.887795 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.888986 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.892122 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:01.894151 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:01.895289 (+ 1138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:01.895464 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:01.895529 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 07:22:01.895640 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5821,"thread_start_us":146,"threads_started":1}
I20260926 07:22:01.899798 10020 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3e8eb4b9df4181803b688f9bde60e3 (DEFAULT_TABLE table=test-table [id=061ba2b6504a4579b2a7935fa29927e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:01.900908 10398 tablet_bootstrap.cc:492] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d: Bootstrap starting.
I20260926 07:22:01.901230 10020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3e8eb4b9df4181803b688f9bde60e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:01.903452 10398 tablet_bootstrap.cc:654] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:01.904392 10398 log.cc:824] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:01.910673 10400 tablet_bootstrap.cc:492] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba: Bootstrap starting.
I20260926 07:22:01.912158 10397 tablet_bootstrap.cc:492] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: Bootstrap starting.
I20260926 07:22:01.913806 10400 tablet_bootstrap.cc:654] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:01.914644 10397 tablet_bootstrap.cc:654] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:01.914808 10400 log.cc:824] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:01.915499 10397 log.cc:824] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:01.918146 10397 tablet_bootstrap.cc:492] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: No bootstrap required, opened a new log
I20260926 07:22:01.918380 10397 ts_tablet_manager.cc:1397] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:22:01.923264 10397 raft_consensus.cc:348] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 } } peers { permanent_uuid: "310f94fb6e1140ffaf8c63134b031d80" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 40451 } } peers { permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 } }
I20260926 07:22:01.923623 10397 raft_consensus.cc:374] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:01.923691 10397 raft_consensus.cc:729] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310f94fb6e1140ffaf8c63134b031d80, State: Initialized, Role: FOLLOWER
I20260926 07:22:01.924297 10397 consensus_queue.cc:260] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 } } peers { permanent_uuid: "310f94fb6e1140ffaf8c63134b031d80" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 40451 } } peers { permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 } }
I20260926 07:22:01.925546 10397 ts_tablet_manager.cc:1428] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 07:22:01.925961 10364 heartbeater.cc:503] Master 127.28.235.254:37009 was elected leader, sending a full tablet report...
I20260926 07:22:01.927615 10398 tablet_bootstrap.cc:492] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d: No bootstrap required, opened a new log
I20260926 07:22:01.927870 10398 ts_tablet_manager.cc:1397] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20260926 07:22:01.933249 10398 raft_consensus.cc:348] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 } } peers { permanent_uuid: "310f94fb6e1140ffaf8c63134b031d80" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 40451 } } peers { permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 } }
I20260926 07:22:01.933677 10398 raft_consensus.cc:374] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:01.933772 10398 raft_consensus.cc:729] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7d50af2e384d9e965649cc9c85430d, State: Initialized, Role: FOLLOWER
I20260926 07:22:01.934788 10398 consensus_queue.cc:260] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 } } peers { permanent_uuid: "310f94fb6e1140ffaf8c63134b031d80" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 40451 } } peers { permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 } }
I20260926 07:22:01.936160 10398 ts_tablet_manager.cc:1428] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260926 07:22:01.936672 10231 heartbeater.cc:503] Master 127.28.235.254:37009 was elected leader, sending a full tablet report...
I20260926 07:22:01.937233 10400 tablet_bootstrap.cc:492] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba: No bootstrap required, opened a new log
I20260926 07:22:01.937463 10400 ts_tablet_manager.cc:1397] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20260926 07:22:01.945981 10400 raft_consensus.cc:348] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 } } peers { permanent_uuid: "310f94fb6e1140ffaf8c63134b031d80" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 40451 } } peers { permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 } }
I20260926 07:22:01.946348 10400 raft_consensus.cc:374] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:01.946408 10400 raft_consensus.cc:729] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81858e64f984b3cb2fcc559e4d8b9ba, State: Initialized, Role: FOLLOWER
I20260926 07:22:01.946931 10400 consensus_queue.cc:260] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 } } peers { permanent_uuid: "310f94fb6e1140ffaf8c63134b031d80" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 40451 } } peers { permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 } }
I20260926 07:22:01.947716 10096 heartbeater.cc:503] Master 127.28.235.254:37009 was elected leader, sending a full tablet report...
I20260926 07:22:01.948154 10400 ts_tablet_manager.cc:1428] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
W20260926 07:22:01.992620 10097 tablet.cc:2369] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:22:02.043560 10232 tablet.cc:2369] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:22:02.100665 10365 tablet.cc:2369] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:22:02.105830 10403 raft_consensus.cc:482] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:22:02.106073 10403 raft_consensus.cc:504] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 } } peers { permanent_uuid: "310f94fb6e1140ffaf8c63134b031d80" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 40451 } } peers { permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 } }
I20260926 07:22:02.107292 10403 leader_election.cc:288] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f81858e64f984b3cb2fcc559e4d8b9ba (127.28.235.193:34065), 7b7d50af2e384d9e965649cc9c85430d (127.28.235.194:45177)
I20260926 07:22:02.113405 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:02.107599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:50753 (local address 127.28.235.193:34065)
0926 07:22:02.107730 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:02.107738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:02.108033 (+ 295us) server_negotiation.cc:407] Connection header received
0926 07:22:02.108529 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:02.108535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:02.108593 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:02.108643 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:02.109094 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:02.109915 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:02.111878 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:02.112485 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:02.112583 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:02.112638 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:02.113157 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 07:22:02.113264 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:22:02.113405 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:02.108287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:36367 (local address 127.28.235.194:45177)
0926 07:22:02.108413 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:02.108419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:02.108435 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:22:02.108588 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:02.108591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:02.108640 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:02.108685 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:02.109095 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:02.109916 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:02.111878 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:02.112485 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:02.112583 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:02.112638 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:02.113212 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 07:22:02.113290 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:22:02.113868 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:02.107903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:45177 (local address 127.28.235.195:36367)
0926 07:22:02.108368 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:02.108380 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:02.108469 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:02.108826 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:02.108829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:02.108867 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:02.108946 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:02.108951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:02.110067 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:02.110070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:02.111717 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:02.111730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:02.112740 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:02.112743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:02.112953 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:02.113005 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:22:02.113573 (+ 568us) client_negotiation.cc:240] Negotiation successful
0926 07:22:02.113690 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":281,"spinlock_wait_cycles":2304,"thread_start_us":91,"threads_started":1}
I20260926 07:22:02.114367 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:02.107508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:34065 (local address 127.28.235.195:50753)
0926 07:22:02.107930 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:02.107947 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:02.108116 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:02.108795 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:02.108798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:02.108817 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:02.108933 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:02.108940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:02.110067 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:02.110070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:02.111717 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:02.111730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:02.112740 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:02.112743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:02.112949 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:02.113006 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 07:22:02.114118 (+ 1112us) client_negotiation.cc:240] Negotiation successful
0926 07:22:02.114196 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":216,"spinlock_wait_cycles":8064,"thread_start_us":99,"threads_started":1}
I20260926 07:22:02.114619 10169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3e8eb4b9df4181803b688f9bde60e3" candidate_uuid: "310f94fb6e1140ffaf8c63134b031d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7d50af2e384d9e965649cc9c85430d" is_pre_election: true
I20260926 07:22:02.114984 10036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3e8eb4b9df4181803b688f9bde60e3" candidate_uuid: "310f94fb6e1140ffaf8c63134b031d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" is_pre_election: true
I20260926 07:22:02.115005 10169 raft_consensus.cc:2454] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 310f94fb6e1140ffaf8c63134b031d80 in term 0.
I20260926 07:22:02.115197 10036 raft_consensus.cc:2454] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 310f94fb6e1140ffaf8c63134b031d80 in term 0.
I20260926 07:22:02.115592 10252 leader_election.cc:302] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 310f94fb6e1140ffaf8c63134b031d80, 7b7d50af2e384d9e965649cc9c85430d; no voters:
I20260926 07:22:02.115926 10403 raft_consensus.cc:2781] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:22:02.116024 10403 raft_consensus.cc:482] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:22:02.116074 10403 raft_consensus.cc:3037] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:02.117251 10403 raft_consensus.cc:504] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 } } peers { permanent_uuid: "310f94fb6e1140ffaf8c63134b031d80" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 40451 } } peers { permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 } }
I20260926 07:22:02.117609 10403 leader_election.cc:288] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [CANDIDATE]: Term 1 election: Requested vote from peers f81858e64f984b3cb2fcc559e4d8b9ba (127.28.235.193:34065), 7b7d50af2e384d9e965649cc9c85430d (127.28.235.194:45177)
I20260926 07:22:02.118481 10169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3e8eb4b9df4181803b688f9bde60e3" candidate_uuid: "310f94fb6e1140ffaf8c63134b031d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7d50af2e384d9e965649cc9c85430d"
I20260926 07:22:02.118626 10169 raft_consensus.cc:3037] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:02.119683 10169 raft_consensus.cc:2454] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 310f94fb6e1140ffaf8c63134b031d80 in term 1.
I20260926 07:22:02.120067 10252 leader_election.cc:302] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 310f94fb6e1140ffaf8c63134b031d80, 7b7d50af2e384d9e965649cc9c85430d; no voters:
I20260926 07:22:02.120301 10403 raft_consensus.cc:2781] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:22:02.120340 10036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3e8eb4b9df4181803b688f9bde60e3" candidate_uuid: "310f94fb6e1140ffaf8c63134b031d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba"
I20260926 07:22:02.120474 10036 raft_consensus.cc:3037] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:02.120529 10403 raft_consensus.cc:686] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 1 LEADER]: Becoming Leader. State: Replica: 310f94fb6e1140ffaf8c63134b031d80, State: Running, Role: LEADER
I20260926 07:22:02.120916 10403 consensus_queue.cc:237] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 } } peers { permanent_uuid: "310f94fb6e1140ffaf8c63134b031d80" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 40451 } } peers { permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 } }
I20260926 07:22:02.121806 10036 raft_consensus.cc:2454] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 310f94fb6e1140ffaf8c63134b031d80 in term 1.
I20260926 07:22:02.125276 9898 catalog_manager.cc:5495] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 310f94fb6e1140ffaf8c63134b031d80 (127.28.235.195). New cstate: current_term: 1 leader_uuid: "310f94fb6e1140ffaf8c63134b031d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "310f94fb6e1140ffaf8c63134b031d80" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 40451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 } health_report { overall_health: UNKNOWN } } }
I20260926 07:22:02.191110 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:02.185432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.28.235.195:40451)
0926 07:22:02.185570 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:02.185575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:02.185592 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:22:02.185693 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:02.185697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:02.185777 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:02.185905 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:22:02.186347 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:02.187158 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:02.188527 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:02.188861 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:02.188967 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:02.189703 (+ 736us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:22:02.190331 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:22:02.190470 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:02.190850 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 07:22:02.190921 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:22:02.199692 10036 raft_consensus.cc:1261] T 0f3e8eb4b9df4181803b688f9bde60e3 P f81858e64f984b3cb2fcc559e4d8b9ba [term 1 FOLLOWER]: Refusing update from remote peer 310f94fb6e1140ffaf8c63134b031d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:02.199692 10169 raft_consensus.cc:1261] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d [term 1 FOLLOWER]: Refusing update from remote peer 310f94fb6e1140ffaf8c63134b031d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:02.200753 10408 consensus_queue.cc:1035] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7d50af2e384d9e965649cc9c85430d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:02.201285 10408 consensus_queue.cc:1035] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f81858e64f984b3cb2fcc559e4d8b9ba" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 34065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:02.210069 10412 mvcc.cc:205] Tried to move back new op lower bound from 7333508391721598976 to 7333508391411273728. Current Snapshot: MvccSnapshot[applied={T|T < 7333508391721598976}]
I20260926 07:22:02.218815 10413 mvcc.cc:205] Tried to move back new op lower bound from 7333508391721598976 to 7333508391411273728. Current Snapshot: MvccSnapshot[applied={T|T < 7333508391721598976}]
I20260926 07:22:02.221657 9893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:22:02.221866 9893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:22:02.221959 10415 mvcc.cc:205] Tried to move back new op lower bound from 7333508391721598976 to 7333508391411273728. Current Snapshot: MvccSnapshot[applied={T|T < 7333508391721598976}]
Sep 26 07:22:02 dist-test-slave-7zlw krb5kdc[8902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407321, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for HTTP/127.28.235.212@KRBTEST.COM
I20260926 07:22:02.227917 9893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P debb0d8c02514b878568a74f3c853f30: Sending DeleteTablet for 3 replicas of tablet 0f3e8eb4b9df4181803b688f9bde60e3
I20260926 07:22:02.228982 10298 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f3e8eb4b9df4181803b688f9bde60e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:02 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:38846
I20260926 07:22:02.231417 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 9936
I20260926 07:22:02.236734 10164 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f3e8eb4b9df4181803b688f9bde60e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:02 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:59372
I20260926 07:22:02.243403 10421 tablet_replica.cc:330] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: stopping tablet replica
I20260926 07:22:02.243937 10421 raft_consensus.cc:2229] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:22:02.244441 10422 tablet_replica.cc:330] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d: stopping tablet replica
I20260926 07:22:02.245280 10421 raft_consensus.cc:2258] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:02.245659 10422 raft_consensus.cc:2229] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:22:02.246060 10422 raft_consensus.cc:2258] T 0f3e8eb4b9df4181803b688f9bde60e3 P 7b7d50af2e384d9e965649cc9c85430d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:02.247776 10421 ts_tablet_manager.cc:1905] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:02.248948 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 10100
W20260926 07:22:02.249533 9883 connection.cc:535] client connection to 127.28.235.193:34065 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 07:22:02.249691 9883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:22:02.252418 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:02.252084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:34065 (local address 127.0.0.1:37460)
0926 07:22:02.252239 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:02.252323 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:34065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:22:02.252626 9883 catalog_manager.cc:4584] TS f81858e64f984b3cb2fcc559e4d8b9ba (127.28.235.193:34065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f3e8eb4b9df4181803b688f9bde60e3: Network error: Client connection negotiation failed: client connection to 127.28.235.193:34065: connect: Connection refused (error 111)
I20260926 07:22:02.253672 10421 ts_tablet_manager.cc:1918] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:22:02.253793 10421 log.cc:1196] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/0f3e8eb4b9df4181803b688f9bde60e3
I20260926 07:22:02.254139 10421 ts_tablet_manager.cc:1939] T 0f3e8eb4b9df4181803b688f9bde60e3 P 310f94fb6e1140ffaf8c63134b031d80: Deleting consensus metadata
I20260926 07:22:02.258611 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 10234
I20260926 07:22:02.262032 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:02.261691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:45177 (local address 127.0.0.1:59376)
0926 07:22:02.261865 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:02.261930 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.194:45177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 07:22:02.265336 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 9865
I20260926 07:22:02.274360 29615 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:22:02.619937 29615 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:22:02.620060 29615 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:22:02.966118 29615 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:22:02.966 UTC [8932] LOG: received smart shutdown request
2026-09-26 07:22:02.970 UTC [8932] LOG: background worker "logical replication launcher" (PID 8941) exited with exit code 1
2026-09-26 07:22:02.970 UTC [8936] LOG: shutting down
2026-09-26 07:22:03.179 UTC [8932] LOG: database system is shut down
2026-09-26T07:22:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52625 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:22:03 dist-test-slave-7zlw krb5kdc[10433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:22:03 dist-test-slave-7zlw krb5kdc[10433](info): set up 2 sockets
Sep 26 07:22:03 dist-test-slave-7zlw krb5kdc[10433](info): commencing operation
krb5kdc: starting...
W20260926 07:22:04.361260 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:22:04 dist-test-slave-7zlw krb5kdc[10433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:22:04.490448 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:41819
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:41819,127.28.235.253:41937,127.28.235.252:35951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:04.608942 10446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:04.609288 10446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:04.609333 10446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:04.613619 10446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:22:04.613704 10446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:04.613725 10446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:04.613744 10446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:22:04.613760 10446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:22:04.619563 10446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:41819,127.28.235.253:41937,127.28.235.252:35951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:04.620793 10446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:04.621951 10446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:04.631203 10451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:04.631203 10452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:04.631234 10454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:04.631567 10446 server_base.cc:1034] running on GCE node
I20260926 07:22:04.632284 10446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:04.633565 10446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:04.634800 10446 hybrid_clock.cc:648] HybridClock initialized: now 1790407324634769 us; error 56 us; skew 500 ppm
Sep 26 07:22:04 dist-test-slave-7zlw krb5kdc[10433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:04.638293 10446 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:22:04.639408 10446 webserver.cc:460] Webserver started at http://127.28.235.254:40809/ using document root <none> and password file <none>
I20260926 07:22:04.639976 10446 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:04.640041 10446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:04.640239 10446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:04.642549 10446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "68bac4aae32346d8a740859e2f6666a0"
format_stamp: "Formatted at 2026-09-26 07:22:04 on dist-test-slave-7zlw"
server_key: "7ca6eb0a28bd760deca2c636d871956a"
server_key_iv: "f527c3e5e6ec00247429b2a23959ae8b"
server_key_version: "encryptionkey@0"
I20260926 07:22:04.643134 10446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "68bac4aae32346d8a740859e2f6666a0"
format_stamp: "Formatted at 2026-09-26 07:22:04 on dist-test-slave-7zlw"
server_key: "7ca6eb0a28bd760deca2c636d871956a"
server_key_iv: "f527c3e5e6ec00247429b2a23959ae8b"
server_key_version: "encryptionkey@0"
I20260926 07:22:04.647248 10446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:22:04.650033 10461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:04.651571 10446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:04.651717 10446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "68bac4aae32346d8a740859e2f6666a0"
format_stamp: "Formatted at 2026-09-26 07:22:04 on dist-test-slave-7zlw"
server_key: "7ca6eb0a28bd760deca2c636d871956a"
server_key_iv: "f527c3e5e6ec00247429b2a23959ae8b"
server_key_version: "encryptionkey@0"
I20260926 07:22:04.651835 10446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:04.662786 10446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:04.666290 10446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:04.666456 10446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:04.673960 10446 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:41819
I20260926 07:22:04.674263 10513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:41819 every 8 connection(s)
I20260926 07:22:04.675040 10446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:22:04.677312 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 10446
I20260926 07:22:04.677409 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:22:04.677716 29615 external_mini_cluster.cc:1428] Setting key 568cc12002975c27c688ec1cf25bbf40
I20260926 07:22:04.678169 10514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:04.682395 10514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.235.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.253@KRBTEST.COM" created.
Sep 26 07:22:04 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:04.700738 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.685525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.28.235.254:41819)
0926 07:22:04.686187 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:04.686204 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:04.686399 (+ 195us) server_negotiation.cc:407] Connection header received
0926 07:22:04.687448 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:04.687453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:04.687635 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:04.687794 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:04.688678 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.689861 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.691185 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.691826 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.692003 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:04.694938 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:04.694950 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:04.694968 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:04.695000 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:04.697253 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:04.697895 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:04.697905 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:04.697908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:04.698065 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:04.698435 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:04.698438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:04.698439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:04.698893 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:04.699038 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:04.699571 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 07:22:04.699799 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"spinlock_wait_cycles":59264,"thread_start_us":222,"threads_started":1}
I20260926 07:22:04.700736 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.685525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:41819 (local address 127.0.0.1:55936)
0926 07:22:04.686247 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.686289 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:04.687159 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:04.687877 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:04.687882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:04.688265 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:04.688520 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:04.688533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.690000 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.690006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:04.691042 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:04.691051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.692005 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.692012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:04.692170 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:04.692832 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:04.692859 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:04.694760 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:04.697380 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:04.697389 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:04.697399 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:04.697729 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:04.698189 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:04.698192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:04.698193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:04.698329 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:04.699087 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:04.699098 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:04.699273 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 07:22:04.699466 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 07:22:04.699902 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":222,"threads_started":1}
I20260926 07:22:04.703431 10514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:22:04.704246 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.703955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:41937 (local address 127.0.0.1:38340)
0926 07:22:04.704094 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.704178 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:41937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 07:22:04.704519 10462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.235.253:41937: connect: Connection refused (error 111)
I20260926 07:22:04.705403 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.705147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:41937 (local address 127.0.0.1:38344)
0926 07:22:04.705271 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.705332 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:41937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 07:22:04.705602 10514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.253:41937: Network error: Client connection negotiation failed: client connection to 127.28.235.253:41937: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab.
Entry for principal kudu/127.28.235.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab.
I20260926 07:22:04.708875 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.253:41937
--webserver_interface=127.28.235.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab
--principal=kudu/127.28.235.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:41819,127.28.235.253:41937,127.28.235.252:35951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:22:04.754931 10514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } attempt: 1
I20260926 07:22:04.756085 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.755639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:41937 (local address 127.0.0.1:38356)
0926 07:22:04.755841 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.755960 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:41937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 07:22:04.757193 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.756873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:41937 (local address 127.0.0.1:38358)
0926 07:22:04.757068 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.757131 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:41937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 07:22:04.757359 10514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.253:41937: Network error: Client connection negotiation failed: client connection to 127.28.235.253:41937: connect: Connection refused (error 111)
I20260926 07:22:04.825551 10514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } attempt: 2
W20260926 07:22:04.826247 10521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:04.826612 10521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260926 07:22:04.826622 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.826237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:41937 (local address 127.0.0.1:38372)
0926 07:22:04.826437 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.826503 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:41937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260926 07:22:04.826686 10521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:22:04.827494 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.827262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:41937 (local address 127.0.0.1:38382)
0926 07:22:04.827384 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.827433 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:41937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 07:22:04.827759 10514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.253:41937: Network error: Client connection negotiation failed: client connection to 127.28.235.253:41937: connect: Connection refused (error 111)
W20260926 07:22:04.831483 10521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:22:04.831560 10521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:04.831581 10521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:04.831598 10521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:22:04.831614 10521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:22:04.837548 10521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:41819,127.28.235.253:41937,127.28.235.252:35951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab
--principal=kudu/127.28.235.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.253:41937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.235.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:04.838763 10521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:04.839874 10521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:04.849278 10526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:04.849263 10529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:04.849263 10527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:04.849656 10521 server_base.cc:1034] running on GCE node
I20260926 07:22:04.850474 10521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:04.851799 10521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:04.853102 10521 hybrid_clock.cc:648] HybridClock initialized: now 1790407324853083 us; error 29 us; skew 500 ppm
Sep 26 07:22:04 dist-test-slave-7zlw krb5kdc[10433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:04.856880 10521 init.cc:377] Logged in from keytab as kudu/127.28.235.253@KRBTEST.COM (short username kudu)
I20260926 07:22:04.857954 10521 webserver.cc:460] Webserver started at http://127.28.235.253:45761/ using document root <none> and password file <none>
I20260926 07:22:04.858521 10521 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:04.858587 10521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:04.858772 10521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:04.861065 10521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/data/instance:
uuid: "8d6beddd5db04ccf9ef23b6af5ef2280"
format_stamp: "Formatted at 2026-09-26 07:22:04 on dist-test-slave-7zlw"
server_key: "919a1f4bc8ffdd6238fc9a785160be54"
server_key_iv: "3c3039e51e5167a096ab072eab568c19"
server_key_version: "encryptionkey@0"
I20260926 07:22:04.861634 10521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/wal/instance:
uuid: "8d6beddd5db04ccf9ef23b6af5ef2280"
format_stamp: "Formatted at 2026-09-26 07:22:04 on dist-test-slave-7zlw"
server_key: "919a1f4bc8ffdd6238fc9a785160be54"
server_key_iv: "3c3039e51e5167a096ab072eab568c19"
server_key_version: "encryptionkey@0"
I20260926 07:22:04.866163 10521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:22:04.869314 10536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:04.870800 10521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:04.870924 10521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/wal
uuid: "8d6beddd5db04ccf9ef23b6af5ef2280"
format_stamp: "Formatted at 2026-09-26 07:22:04 on dist-test-slave-7zlw"
server_key: "919a1f4bc8ffdd6238fc9a785160be54"
server_key_iv: "3c3039e51e5167a096ab072eab568c19"
server_key_version: "encryptionkey@0"
I20260926 07:22:04.871031 10521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:04.898694 10521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:04.902269 10521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:04.902441 10521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:04.910226 10521 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.253:41937
I20260926 07:22:04.910519 10588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.253:41937 every 8 connection(s)
I20260926 07:22:04.911295 10521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
I20260926 07:22:04.914637 10589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:04.916316 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 10521
I20260926 07:22:04.916435 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-1/wal/instance
I20260926 07:22:04.916867 29615 external_mini_cluster.cc:1428] Setting key bbb03561e2d5f74812d6b0527b4a947e
I20260926 07:22:04.918946 10514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } attempt: 3
I20260926 07:22:04.919049 10589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.235.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.252@KRBTEST.COM" created.
Sep 26 07:22:04 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
Sep 26 07:22:04 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
I20260926 07:22:04.935932 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.919552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:41937 (local address 127.0.0.1:38396)
0926 07:22:04.919783 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.919800 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:04.919903 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:04.922526 (+ 2623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:04.922529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:04.922761 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:04.922851 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:04.922856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.926588 (+ 3732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.926593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:04.927415 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:04.927426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.928226 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.928230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:04.928342 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:04.928821 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:04.928833 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:04.931449 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:04.933933 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:04.933936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:04.933939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:04.934197 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:04.934626 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:04.934629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:04.934630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:04.934730 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:04.935362 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:04.935365 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:04.935490 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 07:22:04.935608 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 07:22:04.935763 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":40,"spinlock_wait_cycles":768}
I20260926 07:22:04.936285 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.921416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.28.235.254:41819)
0926 07:22:04.921556 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:04.921562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:04.922060 (+ 498us) server_negotiation.cc:407] Connection header received
0926 07:22:04.922483 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:04.922489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:04.922545 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:04.922628 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:04.923823 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.924593 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.926552 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.926994 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.927150 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:04.931361 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:04.931368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:04.931374 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:04.931411 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:04.934010 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:04.934564 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:04.934568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:04.934569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:04.934659 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:04.935050 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:04.935056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:04.935061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:04.935335 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:04.935418 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:04.935994 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 07:22:04.936112 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":3200}
I20260926 07:22:04.937541 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.921384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:41819 (local address 127.0.0.1:55940)
0926 07:22:04.921895 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.921962 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:04.922272 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:04.922729 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:04.922741 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:04.923264 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:04.923631 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:04.923650 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.924743 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.924748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:04.926383 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:04.926397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.927197 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.927201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:04.927433 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:04.928465 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:04.928509 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:04.931140 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:04.934134 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:04.934146 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:04.934161 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:04.934455 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:04.934758 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:04.934764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:04.934769 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:04.934934 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:04.935513 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:04.935522 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:04.935711 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 07:22:04.935924 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 07:22:04.936370 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":36,"thread_start_us":127,"threads_started":1}
I20260926 07:22:04.937548 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.920314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.28.235.253:41937)
0926 07:22:04.920796 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:04.920814 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:04.920857 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:22:04.921866 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:04.921889 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:04.922270 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:04.922991 (+ 721us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:04.923366 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.924728 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.927535 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.928090 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.928215 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:04.931610 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:04.931617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:04.931631 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:04.931654 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:04.933792 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:04.934329 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:04.934339 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:04.934344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:04.934499 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:04.934812 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:04.934815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:04.934817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:04.935248 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:04.935398 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:04.936681 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0926 07:22:04.936802 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":322,"thread_start_us":175,"threads_started":1}
I20260926 07:22:04.940513 10514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:22:04.941407 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.941109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50236)
0926 07:22:04.941281 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.941332 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 07:22:04.946120 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.945736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50252)
0926 07:22:04.945983 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.946039 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260926 07:22:04.946373 10514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:35951: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
I20260926 07:22:04.946729 10589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab.
Entry for principal kudu/127.28.235.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab.
Sep 26 07:22:04 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
I20260926 07:22:04.953732 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.252:35951
--webserver_interface=127.28.235.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab
--principal=kudu/127.28.235.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:41819,127.28.235.253:41937,127.28.235.252:35951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:22:04.956905 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.947300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:41937 (local address 127.0.0.1:38398)
0926 07:22:04.948031 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.948045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:04.948154 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:04.948444 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:04.948449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:04.948626 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:04.948726 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:04.948732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.949684 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.949688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:04.950339 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:04.950349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.950860 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.950863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:04.950941 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:04.951592 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:04.951606 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:04.953218 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:04.955230 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:04.955235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:04.955241 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:04.955492 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:04.955780 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:04.955785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:04.955787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:04.955877 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:04.956296 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:04.956300 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:04.956394 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 07:22:04.956520 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 07:22:04.956676 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633}
I20260926 07:22:04.956905 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.947396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.28.235.253:41937)
0926 07:22:04.947518 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:04.947522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:04.948128 (+ 606us) server_negotiation.cc:407] Connection header received
0926 07:22:04.948242 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:04.948245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:04.948302 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:04.948388 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:04.948896 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.949548 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.950475 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:04.950736 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:04.950829 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:04.953369 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:04.953373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:04.953375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:04.953400 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:04.955056 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:04.955579 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:04.955585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:04.955587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:04.955668 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:04.955979 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:04.955983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:04.955985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:04.956183 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:04.956310 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:04.956625 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 07:22:04.956729 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 07:22:04.957943 10589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:22:04.958844 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.958497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50264)
0926 07:22:04.958637 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.958741 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 07:22:04.959751 10539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
I20260926 07:22:04.961100 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.960784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50268)
0926 07:22:04.960910 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.960969 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:22:04.961282 10589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:35951: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
I20260926 07:22:04.977558 10514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } attempt: 1
I20260926 07:22:04.978583 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.978208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50280)
0926 07:22:04.978401 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.978496 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 07:22:04.979532 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:04.979261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50286)
0926 07:22:04.979417 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:04.979469 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 07:22:04.979785 10514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:35951: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
I20260926 07:22:05.010653 10589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } attempt: 1
I20260926 07:22:05.011675 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.011354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50298)
0926 07:22:05.011523 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.011592 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 07:22:05.012467 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.012235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50314)
0926 07:22:05.012352 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.012407 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 07:22:05.012624 10589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:35951: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
I20260926 07:22:05.055037 10514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } attempt: 2
I20260926 07:22:05.056155 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.055743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50328)
0926 07:22:05.055987 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.056054 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260926 07:22:05.057180 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.056896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50340)
0926 07:22:05.057048 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.057116 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:22:05.057354 10514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:35951: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
I20260926 07:22:05.080830 10589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } attempt: 2
I20260926 07:22:05.081768 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.081403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50354)
0926 07:22:05.081592 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.081671 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 07:22:05.082126 10595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:05.082391 10595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:05.082468 10595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:22:05.082732 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.082475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50368)
0926 07:22:05.082609 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.082667 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:22:05.082921 10589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:35951: Network error: Client connection negotiation failed: client connection to 127.28.235.252:35951: connect: Connection refused (error 111)
W20260926 07:22:05.086978 10595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:22:05.087049 10595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:05.087069 10595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:05.087086 10595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:22:05.087143 10595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:22:05.092988 10595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:41819,127.28.235.253:41937,127.28.235.252:35951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab
--principal=kudu/127.28.235.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.252:35951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.235.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:05.094285 10595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:05.095374 10595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:05.104598 10602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:05.104600 10604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:05.104785 10595 server_base.cc:1034] running on GCE node
W20260926 07:22:05.105100 10601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:05.105731 10595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:05.106980 10595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:05.108202 10595 hybrid_clock.cc:648] HybridClock initialized: now 1790407325108170 us; error 49 us; skew 500 ppm
Sep 26 07:22:05 dist-test-slave-7zlw krb5kdc[10433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:05.111573 10595 init.cc:377] Logged in from keytab as kudu/127.28.235.252@KRBTEST.COM (short username kudu)
I20260926 07:22:05.112560 10595 webserver.cc:460] Webserver started at http://127.28.235.252:36437/ using document root <none> and password file <none>
I20260926 07:22:05.113293 10595 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:05.113389 10595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:05.113639 10595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:05.115833 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/data/instance:
uuid: "c5efa737149c4d8a86a6582c31baba95"
format_stamp: "Formatted at 2026-09-26 07:22:05 on dist-test-slave-7zlw"
server_key: "0771f4299f09c1fa0a6f78c10c4de785"
server_key_iv: "38359cc707e515f3d1374c6385ae1e21"
server_key_version: "encryptionkey@0"
I20260926 07:22:05.116500 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/wal/instance:
uuid: "c5efa737149c4d8a86a6582c31baba95"
format_stamp: "Formatted at 2026-09-26 07:22:05 on dist-test-slave-7zlw"
server_key: "0771f4299f09c1fa0a6f78c10c4de785"
server_key_iv: "38359cc707e515f3d1374c6385ae1e21"
server_key_version: "encryptionkey@0"
I20260926 07:22:05.121647 10595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:22:05.124543 10611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:05.126121 10595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:05.126292 10595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/wal
uuid: "c5efa737149c4d8a86a6582c31baba95"
format_stamp: "Formatted at 2026-09-26 07:22:05 on dist-test-slave-7zlw"
server_key: "0771f4299f09c1fa0a6f78c10c4de785"
server_key_iv: "38359cc707e515f3d1374c6385ae1e21"
server_key_version: "encryptionkey@0"
I20260926 07:22:05.126428 10595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:05.141588 10595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:05.144985 10595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:05.145265 10595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:05.153128 10595 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.252:35951
I20260926 07:22:05.153496 10663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.252:35951 every 8 connection(s)
I20260926 07:22:05.154245 10595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
I20260926 07:22:05.156527 10514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } attempt: 3
I20260926 07:22:05.157809 10664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:05.161252 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 10595
I20260926 07:22:05.161386 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407154183638-29615-0/minicluster-data/master-2/wal/instance
I20260926 07:22:05.161681 29615 external_mini_cluster.cc:1428] Setting key 2d5bde03b523ebd0204552eb2667cdaf
I20260926 07:22:05.162930 10664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:22:05 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
Sep 26 07:22:05 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
I20260926 07:22:05.171706 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.157218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50374)
0926 07:22:05.157361 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.157378 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:05.157475 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:05.159873 (+ 2398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:05.159876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:05.160108 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:05.160252 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:05.160262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.162133 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.162137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:05.162923 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:05.162935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.163829 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.163833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:05.163925 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.164557 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:05.164570 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:05.166176 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:05.169138 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:05.169142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:05.169144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:05.169372 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:05.169802 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:05.169806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:05.169808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:05.169940 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:05.170783 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:05.170789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:05.170922 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 07:22:05.171051 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 07:22:05.171481 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
Sep 26 07:22:05 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:05.172854 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.157998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.28.235.252:35951)
0926 07:22:05.158479 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.158496 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.158535 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:22:05.159205 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.159221 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.159564 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.159916 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.160497 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.161961 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.163056 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.163679 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.163876 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.166391 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.166401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.166421 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.166452 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.168992 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.169512 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.169520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.169528 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.169685 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.170048 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.170052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.170054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.170653 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.170831 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.171213 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.171621 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":172,"threads_started":1}
I20260926 07:22:05.174252 10589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } attempt: 3
I20260926 07:22:05.174497 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.162382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.28.235.252:35951)
0926 07:22:05.162731 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.162739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.163237 (+ 498us) server_negotiation.cc:407] Connection header received
0926 07:22:05.163276 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.163279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.163327 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.163417 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.164423 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.165220 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.166470 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.166848 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.166958 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.170075 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.170079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.170081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.170103 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.172154 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.172621 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.172626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.172628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.172721 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.173072 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.173075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.173077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.173325 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.173513 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.174164 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.174278 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
I20260926 07:22:05.177577 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.165327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.28.235.254:41819)
0926 07:22:05.165454 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.165458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.166314 (+ 856us) server_negotiation.cc:407] Connection header received
0926 07:22:05.166453 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.166457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.166505 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.166577 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.167555 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.168276 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.169444 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.169824 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.169941 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.173312 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.173316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.173318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.173351 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.175163 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.175801 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.175805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.175807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.176171 (+ 364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.176516 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.176520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.176522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.176793 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.176852 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.177297 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.177423 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:22:05.178409 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.165292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:41819 (local address 127.0.0.1:55946)
0926 07:22:05.166157 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.166198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:05.166345 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:05.166649 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:05.166657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:05.167151 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:05.167289 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:05.167300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.168390 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.168397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:05.169301 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:05.169314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.169946 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.169950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:05.170068 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.170711 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:05.170730 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:05.173160 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:05.175310 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:05.175317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:05.175323 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:05.175685 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:05.176308 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:05.176311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:05.176313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:05.176404 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:05.176920 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:05.176925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:05.177080 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 07:22:05.177246 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 07:22:05.178225 (+ 979us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"spinlock_wait_cycles":6016,"thread_start_us":101,"threads_started":1}
I20260926 07:22:05.178560 10514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0: Bootstrap starting.
I20260926 07:22:05.179601 10664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:22:05 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
I20260926 07:22:05.182353 10514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:05.183457 10514 log.cc:824] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:22:05 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:05.185467 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.174686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50388)
0926 07:22:05.174945 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.174959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:05.175037 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:05.175540 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:05.175543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:05.175747 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:05.175879 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:05.175886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.178115 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.178119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:05.178937 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:05.178948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.179627 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.179630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:05.179713 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.180269 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:05.180284 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:05.181860 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:05.183712 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:05.183715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:05.183718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:05.183971 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:05.184257 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:05.184260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:05.184262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:05.184361 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:05.184863 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:05.184866 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:05.184972 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 07:22:05.185114 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 07:22:05.185262 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260926 07:22:05.185742 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.175159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.28.235.252:35951)
0926 07:22:05.175282 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.175286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.175300 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:22:05.175333 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.175336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.175375 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.177172 (+ 1797us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.177265 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.177983 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.179086 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.179409 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.179505 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.182003 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.182007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.182010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.182032 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.183613 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.184069 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.184073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.184075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.184155 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.184475 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.184479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.184481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.184708 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.184826 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.185167 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.185274 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":54}
I20260926 07:22:05.186367 10514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0: No bootstrap required, opened a new log
I20260926 07:22:05.188907 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.176389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.28.235.254:41819)
0926 07:22:05.177474 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.177483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.177495 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:22:05.177528 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.177530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.177570 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.177652 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.178522 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.179300 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.180741 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.181127 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.181221 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.185101 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.185105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.185108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.185133 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.186848 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.187335 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.187339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.187341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.187437 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.187760 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.187764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.187765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.187987 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.188178 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.188419 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.188550 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":87,"threads_started":1}
I20260926 07:22:05.189483 10589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280: Bootstrap starting.
I20260926 07:22:05.190431 10514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } }
I20260926 07:22:05.190616 10514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:05.190676 10514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bac4aae32346d8a740859e2f6666a0, State: Initialized, Role: FOLLOWER
I20260926 07:22:05.191114 10514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } }
I20260926 07:22:05.191641 10673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } } }
I20260926 07:22:05.191782 10673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:22:05.192276 10589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:05.192433 10514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:22:05.193147 10589 log.cc:824] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:22:05 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
I20260926 07:22:05.196038 10589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280: No bootstrap required, opened a new log
I20260926 07:22:05.199443 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.180399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.28.235.253:41937)
0926 07:22:05.180524 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.180529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.180628 (+ 99us) server_negotiation.cc:407] Connection header received
0926 07:22:05.180804 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.180810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.180864 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.180916 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.181540 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.182282 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.183478 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.183832 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.183912 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.195544 (+ 11632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.195548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.195551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.195576 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.197505 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.197987 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.197991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.197992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.198077 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.198371 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.198376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.198378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.198592 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.198725 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.199038 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.199227 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:22:05.199913 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.180049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:41937 (local address 127.0.0.1:38414)
0926 07:22:05.180561 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.180576 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:05.180692 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:05.181096 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:05.181099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:05.181290 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:05.181402 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:05.181409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.182436 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.182439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:05.183329 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:05.183343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.184066 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.184069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:05.184179 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.193194 (+ 9015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:05.193216 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:05.195359 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:05.197633 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:05.197637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:05.197640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:05.197867 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:05.198178 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:05.198181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:05.198183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:05.198274 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:05.198714 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:05.198716 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:05.198832 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 07:22:05.199577 (+ 745us) client_negotiation.cc:240] Negotiation successful
0926 07:22:05.199739 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":149,"threads_started":1}
I20260926 07:22:05.200343 10589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } }
I20260926 07:22:05.200522 10664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:22:05.200573 10589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:05.200632 10589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6beddd5db04ccf9ef23b6af5ef2280, State: Initialized, Role: FOLLOWER
I20260926 07:22:05.201232 10589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } }
I20260926 07:22:05.201754 10687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } } }
I20260926 07:22:05.201884 10687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:22:05.198765 10446 master_runner.cc:186] Error getting master registration for 127.28.235.253:41937: OK, instance_id { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" instance_seqno: 1790407324908101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:22:05.202581 10446 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:22:05.202766 10589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 07:22:05 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
Sep 26 07:22:05 dist-test-slave-7zlw krb5kdc[10433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
W20260926 07:22:05.208446 10685 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:22:05.208518 10685 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:22:05.210618 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.194928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.28.235.253:41937)
0926 07:22:05.195221 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.195232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.195249 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:22:05.195292 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.195294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.195349 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.196840 (+ 1491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.196960 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.198615 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.199861 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.200551 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.202457 (+ 1906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.203716 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.203722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.203728 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.203759 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.206621 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.207470 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.207475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.207479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.207577 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.207920 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.207923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.207925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.208148 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.210017 (+ 1869us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.210095 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.210458 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":110,"threads_started":1}
I20260926 07:22:05.213655 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.202955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.28.235.252:35951)
0926 07:22:05.203061 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.203066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.203569 (+ 503us) server_negotiation.cc:407] Connection header received
0926 07:22:05.203712 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.203715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.203752 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.203823 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.204378 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.205240 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.206275 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.206590 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.206663 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.208939 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.208942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.208945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.208966 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.211958 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.212437 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.212441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.212443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.212516 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.212778 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.212781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.212783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.212992 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.213089 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.213379 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.213496 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:22:05.214089 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.202868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:35951 (local address 127.0.0.1:50398)
0926 07:22:05.203490 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:05.203503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:05.203616 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:05.203933 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:05.203937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:05.204124 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:05.204241 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:05.204250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.205369 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.205372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:05.206153 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:05.206164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.206799 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.206802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:05.206871 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.207294 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:05.207315 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:05.208807 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:05.212074 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:05.212077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:05.212080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:05.212333 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:05.212612 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:05.212615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:05.212617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:05.212685 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:05.213144 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:05.213147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:05.213227 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 07:22:05.213815 (+ 588us) client_negotiation.cc:240] Negotiation successful
0926 07:22:05.213949 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555}
I20260926 07:22:05.214658 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:22:05.211215 10521 master_runner.cc:186] Error getting master registration for 127.28.235.252:35951: OK, instance_id { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" instance_seqno: 1790407325150839 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:22:05.215754 10521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:22:05.218622 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95: Bootstrap starting.
I20260926 07:22:05.221812 10664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:05.222863 10664 log.cc:824] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95: Log is configured to *not* fsync() on all Append() calls
W20260926 07:22:05.224475 10698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:22:05.224560 10698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:22:05.226126 10687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:22:05.226334 10687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } }
I20260926 07:22:05.226769 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95: No bootstrap required, opened a new log
I20260926 07:22:05.227846 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68bac4aae32346d8a740859e2f6666a0" is_pre_election: true
I20260926 07:22:05.228158 10488 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6beddd5db04ccf9ef23b6af5ef2280 in term 0.
I20260926 07:22:05.228268 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.217492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.28.235.253:41937)
0926 07:22:05.217623 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.217628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.217928 (+ 300us) server_negotiation.cc:407] Connection header received
0926 07:22:05.218110 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.218114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.218165 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.218214 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.219210 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.219973 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.221136 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.221472 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.221550 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.223500 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.223504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.223506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.223533 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.225815 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.226855 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.226859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.226861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.226949 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.227266 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.227269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.227271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.227495 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.227597 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.227979 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.228110 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:22:05.228669 10538 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68bac4aae32346d8a740859e2f6666a0, 8d6beddd5db04ccf9ef23b6af5ef2280; no voters:
I20260926 07:22:05.230428 10687 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68bac4aae32346d8a740859e2f6666a0 (127.28.235.254:41819), c5efa737149c4d8a86a6582c31baba95 (127.28.235.252:35951)
I20260926 07:22:05.230638 10687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:22:05.230783 10687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:22:05.230901 10687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:05.231395 10664 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } }
I20260926 07:22:05.231683 10664 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:05.231755 10664 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5efa737149c4d8a86a6582c31baba95, State: Initialized, Role: FOLLOWER
I20260926 07:22:05.232693 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.217196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.28.235.252:35951)
0926 07:22:05.217324 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.217329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.221349 (+ 4020us) server_negotiation.cc:407] Connection header received
0926 07:22:05.221527 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.221530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.221582 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.221630 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.222609 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.223355 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.224469 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.224819 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.224921 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.228724 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.228729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.228732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.228758 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.230690 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.231258 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.231261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.231263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.231362 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.231679 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.231683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.231685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.231904 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.232040 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.232420 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.232540 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:22:05.233464 10687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } }
I20260926 07:22:05.234581 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68bac4aae32346d8a740859e2f6666a0"
I20260926 07:22:05.234735 10488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:05.236323 10488 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6beddd5db04ccf9ef23b6af5ef2280 in term 1.
I20260926 07:22:05.236552 10687 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [CANDIDATE]: Term 1 election: Requested vote from peers 68bac4aae32346d8a740859e2f6666a0 (127.28.235.254:41819), c5efa737149c4d8a86a6582c31baba95 (127.28.235.252:35951)
I20260926 07:22:05.236745 10538 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68bac4aae32346d8a740859e2f6666a0, 8d6beddd5db04ccf9ef23b6af5ef2280; no voters:
I20260926 07:22:05.237200 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:05.218060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.28.235.254:41819)
0926 07:22:05.218173 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:05.218177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:05.224804 (+ 6627us) server_negotiation.cc:407] Connection header received
0926 07:22:05.224896 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:05.224899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:05.224958 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:05.225083 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:05.226643 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.227391 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.229957 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:05.230306 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:05.230415 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:05.232687 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:05.232693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:05.232696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:05.232730 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:05.234832 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.235533 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.235537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.235540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.235634 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:05.235977 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:05.235981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:05.235982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:05.236225 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:05.236334 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:05.236846 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 07:22:05.236984 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:22:05.237396 10709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:22:05.237524 10709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [term 1 LEADER]: Becoming Leader. State: Replica: 8d6beddd5db04ccf9ef23b6af5ef2280, State: Running, Role: LEADER
I20260926 07:22:05.237869 10709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } }
I20260926 07:22:05.241139 10687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6beddd5db04ccf9ef23b6af5ef2280. Latest consensus state: current_term: 1 leader_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } } }
I20260926 07:22:05.241302 10687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:05.241729 10711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:22:05.242802 10711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:22:05.249420 10664 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } }
I20260926 07:22:05.250108 10713 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } } }
I20260926 07:22:05.250353 10488 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [term 1 FOLLOWER]: Refusing update from remote peer 8d6beddd5db04ccf9ef23b6af5ef2280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:05.250542 10713 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:22:05.251055 10664 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:22:05.251245 10687 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:05.251364 10638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5efa737149c4d8a86a6582c31baba95" is_pre_election: true
I20260926 07:22:05.251662 10638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6beddd5db04ccf9ef23b6af5ef2280 in term 0.
I20260926 07:22:05.251752 10637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5efa737149c4d8a86a6582c31baba95"
I20260926 07:22:05.251848 10637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:05.253255 10637 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6beddd5db04ccf9ef23b6af5ef2280 in term 1.
I20260926 07:22:05.253770 10636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [term 1 FOLLOWER]: Refusing update from remote peer 8d6beddd5db04ccf9ef23b6af5ef2280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:05.254437 10687 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:05.258303 10713 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6beddd5db04ccf9ef23b6af5ef2280. Latest consensus state: current_term: 1 leader_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } } }
I20260926 07:22:05.258435 10713 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:22:05.260804 10673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6beddd5db04ccf9ef23b6af5ef2280. Latest consensus state: current_term: 1 leader_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } } }
I20260926 07:22:05.260931 10673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:22:05.261991 10687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } } }
I20260926 07:22:05.262126 10687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:05.264068 10711 catalog_manager.cc:1347] Generated new cluster ID: b021f0ea11bd45dbb7714d4d38ff6b1b
I20260926 07:22:05.264153 10711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:22:05.265061 10673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } } }
I20260926 07:22:05.265204 10673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:22:05.265755 10712 mvcc.cc:205] Tried to move back new op lower bound from 7333508404214747136 to 7333508404179632128. Current Snapshot: MvccSnapshot[applied={T|T < 7333508404214747136 or (T in {7333508404214747136})}]
I20260926 07:22:05.267359 10687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } } }
I20260926 07:22:05.267490 10687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280 [sys.catalog]: This master's current role is: LEADER
W20260926 07:22:05.268143 10724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:22:05.268229 10724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:22:05.268626 10713 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68bac4aae32346d8a740859e2f6666a0" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 41819 } } peers { permanent_uuid: "8d6beddd5db04ccf9ef23b6af5ef2280" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 41937 } } peers { permanent_uuid: "c5efa737149c4d8a86a6582c31baba95" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 35951 } } }
I20260926 07:22:05.268759 10713 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:22:05.286490 10711 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:22:05.287832 10711 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:22:05.299245 10711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d6beddd5db04ccf9ef23b6af5ef2280: Generated new TSK 0
I20260926 07:22:05.300228 10711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:22:06.209925 10685 catalog_manager.cc:1259] Loaded cluster ID: b021f0ea11bd45dbb7714d4d38ff6b1b
I20260926 07:22:06.210013 10685 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0: loading cluster ID for follower catalog manager: success
I20260926 07:22:06.211647 10685 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0: acquiring CA information for follower catalog manager: success
I20260926 07:22:06.212818 10685 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68bac4aae32346d8a740859e2f6666a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:22:06.270020 10724 catalog_manager.cc:1259] Loaded cluster ID: b021f0ea11bd45dbb7714d4d38ff6b1b
I20260926 07:22:06.270107 10724 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95: loading cluster ID for follower catalog manager: success
I20260926 07:22:06.271868 10724 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95: acquiring CA information for follower catalog manager: success
I20260926 07:22:06.273195 10724 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5efa737149c4d8a86a6582c31baba95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:22:06.376298 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 10446
I20260926 07:22:06.383090 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 10521
I20260926 07:22:06.389499 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 10595
2026-09-26T07:22:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3095 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:22:06 dist-test-slave-7zlw krb5kdc[10740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:22:06 dist-test-slave-7zlw krb5kdc[10740](info): set up 2 sockets
Sep 26 07:22:06 dist-test-slave-7zlw krb5kdc[10740](info): commencing operation
krb5kdc: starting...
W20260926 07:22:07.457489 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:22:07 dist-test-slave-7zlw krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:22:07.588793 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:46479
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:07.702443 10753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:07.702735 10753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:07.702778 10753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:07.707073 10753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:22:07.707144 10753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:07.707162 10753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:07.707180 10753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:22:07.707222 10753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:22:07.712677 10753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:46479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:07.713934 10753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:07.715047 10753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:07.724442 10758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:07.724452 10761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:07.724445 10759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:07.724786 10753 server_base.cc:1034] running on GCE node
I20260926 07:22:07.725533 10753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:07.726752 10753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:07.727958 10753 hybrid_clock.cc:648] HybridClock initialized: now 1790407327727930 us; error 52 us; skew 500 ppm
Sep 26 07:22:07 dist-test-slave-7zlw krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:07.730926 10753 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:22:07.731901 10753 webserver.cc:460] Webserver started at http://127.28.235.254:44801/ using document root <none> and password file <none>
I20260926 07:22:07.732494 10753 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:07.732590 10753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:07.732838 10753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:07.735061 10753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "8f95d19f5c644780953eccd53c4cf442"
format_stamp: "Formatted at 2026-09-26 07:22:07 on dist-test-slave-7zlw"
server_key: "8a4ab85cead60858117e10becf653345"
server_key_iv: "3d52cb84d34afb7deb57752cfc634c7b"
server_key_version: "encryptionkey@0"
I20260926 07:22:07.735656 10753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "8f95d19f5c644780953eccd53c4cf442"
format_stamp: "Formatted at 2026-09-26 07:22:07 on dist-test-slave-7zlw"
server_key: "8a4ab85cead60858117e10becf653345"
server_key_iv: "3d52cb84d34afb7deb57752cfc634c7b"
server_key_version: "encryptionkey@0"
I20260926 07:22:07.739713 10753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 07:22:07.742687 10768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:07.744061 10753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:22:07.744243 10753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "8f95d19f5c644780953eccd53c4cf442"
format_stamp: "Formatted at 2026-09-26 07:22:07 on dist-test-slave-7zlw"
server_key: "8a4ab85cead60858117e10becf653345"
server_key_iv: "3d52cb84d34afb7deb57752cfc634c7b"
server_key_version: "encryptionkey@0"
I20260926 07:22:07.744433 10753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:07.760568 10753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:07.764055 10753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:07.764294 10753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:07.771872 10753 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:46479
I20260926 07:22:07.771926 10820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:46479 every 8 connection(s)
I20260926 07:22:07.772943 10753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:22:07.775746 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 10753
I20260926 07:22:07.775943 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:22:07.776002 10821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:07.776255 29615 external_mini_cluster.cc:1428] Setting key a0609276c0fc22723b543a94e54f196f
I20260926 07:22:07.783659 10821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442: Bootstrap starting.
Sep 26 07:22:07 dist-test-slave-7zlw krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:07.786604 10821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:07.787642 10821 log.cc:824] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:07.790164 10821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442: No bootstrap required, opened a new log
I20260926 07:22:07.792163 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:07.777678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.28.235.254:46479)
0926 07:22:07.778103 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:07.778119 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:07.778155 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:22:07.778964 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:07.778984 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:07.779343 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:07.779639 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:07.780613 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:07.781767 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:07.782990 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:07.783513 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:07.783640 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:07.786504 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:07.786514 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:07.786533 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:07.786573 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:07.788563 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:07.789040 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:07.789047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:07.789054 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:07.789195 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:07.789503 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:07.789505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:07.789507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:07.790038 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:07.790248 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:07.790583 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:22:07.790912 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":151,"threads_started":1}
I20260926 07:22:07.794270 10821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f95d19f5c644780953eccd53c4cf442" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46479 } }
I20260926 07:22:07.794541 10821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:07.794618 10821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f95d19f5c644780953eccd53c4cf442, State: Initialized, Role: FOLLOWER
I20260926 07:22:07.795128 10821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f95d19f5c644780953eccd53c4cf442" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46479 } }
I20260926 07:22:07.795292 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:22:07.795322 10821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:22:07.795387 10821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:22:07.795481 10821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:07.796501 10821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f95d19f5c644780953eccd53c4cf442" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46479 } }
I20260926 07:22:07.796871 10821 leader_election.cc:302] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f95d19f5c644780953eccd53c4cf442; no voters:
I20260926 07:22:07.797268 10821 leader_election.cc:288] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:22:07.797461 10826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:22:07.797770 10826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [term 1 LEADER]: Becoming Leader. State: Replica: 8f95d19f5c644780953eccd53c4cf442, State: Running, Role: LEADER
I20260926 07:22:07.798072 10826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f95d19f5c644780953eccd53c4cf442" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46479 } }
I20260926 07:22:07.798584 10821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:22:07.799682 10828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f95d19f5c644780953eccd53c4cf442. Latest consensus state: current_term: 1 leader_uuid: "8f95d19f5c644780953eccd53c4cf442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f95d19f5c644780953eccd53c4cf442" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46479 } } }
I20260926 07:22:07.799826 10828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:07.800122 10827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f95d19f5c644780953eccd53c4cf442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f95d19f5c644780953eccd53c4cf442" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46479 } } }
I20260926 07:22:07.800225 10827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:07.800297 10835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:22:07.803670 10835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:22:07.809854 10835 catalog_manager.cc:1347] Generated new cluster ID: 93f529bc2e66403e81530c5e82b1fdde
I20260926 07:22:07.809930 10835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:22:07.844748 10835 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:22:07.846171 10835 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:22:07.856227 10835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f95d19f5c644780953eccd53c4cf442: Generated new TSK 0
I20260926 07:22:07.857203 10835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:22:07.903066 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 10753
2026-09-26T07:22:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1511 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:22:07 dist-test-slave-7zlw krb5kdc[10848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:22:07 dist-test-slave-7zlw krb5kdc[10848](info): set up 2 sockets
Sep 26 07:22:07 dist-test-slave-7zlw krb5kdc[10848](info): commencing operation
krb5kdc: starting...
W20260926 07:22:08.945696 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:22:09 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:22:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:22:09.081938 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:42613
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:42613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:09.198143 10861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:09.198483 10861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:09.198576 10861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:09.202720 10861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:22:09.202828 10861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:09.202867 10861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:09.202903 10861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:22:09.202934 10861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:22:09.208704 10861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:42613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:42613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:09.210106 10861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:09.211323 10861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:09.220566 10866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:09.220600 10869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:09.220678 10861 server_base.cc:1034] running on GCE node
W20260926 07:22:09.220556 10867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:09.221417 10861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:09.222667 10861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:09.223905 10861 hybrid_clock.cc:648] HybridClock initialized: now 1790407329223888 us; error 29 us; skew 500 ppm
Sep 26 07:22:09 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:09.231424 10861 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:22:09.232420 10861 webserver.cc:460] Webserver started at http://127.28.235.254:38725/ using document root <none> and password file <none>
I20260926 07:22:09.232980 10861 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:09.233109 10861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:09.233358 10861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:09.235462 10861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "0cf06e20743d46ea86f0f7e477217cf3"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "c93217f68301030105b47b087a031ebb"
server_key_iv: "82122f06b6e21b5eea03ec9074d44897"
server_key_version: "encryptionkey@0"
I20260926 07:22:09.236042 10861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "0cf06e20743d46ea86f0f7e477217cf3"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "c93217f68301030105b47b087a031ebb"
server_key_iv: "82122f06b6e21b5eea03ec9074d44897"
server_key_version: "encryptionkey@0"
I20260926 07:22:09.240170 10861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 07:22:09.242836 10876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:09.244082 10861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:22:09.244241 10861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "0cf06e20743d46ea86f0f7e477217cf3"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "c93217f68301030105b47b087a031ebb"
server_key_iv: "82122f06b6e21b5eea03ec9074d44897"
server_key_version: "encryptionkey@0"
I20260926 07:22:09.244361 10861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:09.266409 10861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:09.269838 10861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:09.270066 10861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:09.277112 10861 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:42613
I20260926 07:22:09.277412 10928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:42613 every 8 connection(s)
I20260926 07:22:09.278129 10861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:22:09.278990 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 10861
I20260926 07:22:09.279083 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:22:09.279405 29615 external_mini_cluster.cc:1428] Setting key e3183ddca92b292b2f9e512250293491
I20260926 07:22:09.281296 10929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:09.287894 10929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Bootstrap starting.
Sep 26 07:22:09 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:09.290691 10929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:09.291456 10929 log.cc:824] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:09.293643 10929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: No bootstrap required, opened a new log
I20260926 07:22:09.294687 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:09.280835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.28.235.254:42613)
0926 07:22:09.281288 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:09.281309 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:09.281354 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:22:09.282196 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:09.282218 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:09.282614 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:09.282983 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:09.283630 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:09.285169 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:09.286031 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:09.286733 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:09.286914 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:09.289763 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:09.289768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:09.289781 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:09.289809 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:09.291669 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:09.292182 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:09.292188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:09.292195 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:09.292315 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:09.292648 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:09.292651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:09.292653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:09.293135 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:09.293346 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:09.293636 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 07:22:09.293954 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":162,"threads_started":1}
I20260926 07:22:09.297281 10929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:09.297571 10929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:09.297621 10929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf06e20743d46ea86f0f7e477217cf3, State: Initialized, Role: FOLLOWER
I20260926 07:22:09.297984 10929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:09.298094 10929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:22:09.298130 10929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:22:09.298198 10929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:09.299062 10929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:09.299347 10929 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf06e20743d46ea86f0f7e477217cf3; no voters:
I20260926 07:22:09.299613 10929 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:22:09.299774 10934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:22:09.299995 10934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf06e20743d46ea86f0f7e477217cf3, State: Running, Role: LEADER
I20260926 07:22:09.300307 10934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:09.300896 10929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:22:09.302021 10935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cf06e20743d46ea86f0f7e477217cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } } }
I20260926 07:22:09.302150 10935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:09.302459 10936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf06e20743d46ea86f0f7e477217cf3. Latest consensus state: current_term: 1 leader_uuid: "0cf06e20743d46ea86f0f7e477217cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } } }
I20260926 07:22:09.302551 10936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:09.302629 10943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:22:09.305941 10943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:22:09.312093 10943 catalog_manager.cc:1347] Generated new cluster ID: b13c0a8bf98f445583a863a4141052b9
I20260926 07:22:09.312181 10943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:22:09.320704 10943 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:22:09.321774 10943 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:22:09.329985 10943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Generated new TSK 0
I20260926 07:22:09.330888 10943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:22:09.364607 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:42613
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:09.495949 10955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:09.496230 10955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:09.496274 10955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:09.500377 10955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:09.500447 10955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:09.500553 10955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:22:09.506373 10955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:09.507579 10955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:09.508651 10955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:09.518658 10961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:09.518661 10963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:09.518658 10960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:09.518993 10955 server_base.cc:1034] running on GCE node
I20260926 07:22:09.519512 10955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:09.520169 10955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:09.521354 10955 hybrid_clock.cc:648] HybridClock initialized: now 1790407329521341 us; error 35 us; skew 500 ppm
Sep 26 07:22:09 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:09.524417 10955 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:22:09.525326 10955 webserver.cc:460] Webserver started at http://127.28.235.193:33015/ using document root <none> and password file <none>
I20260926 07:22:09.525875 10955 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:09.525933 10955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:09.526099 10955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:09.528139 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "f75c23e2671246f78a4fcbf88d48a502"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "9f331d7d686a4facd8f8b86c68a6df09"
server_key_iv: "7b832d5fb05e5a4a30312deacfb9a927"
server_key_version: "encryptionkey@0"
I20260926 07:22:09.528637 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "f75c23e2671246f78a4fcbf88d48a502"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "9f331d7d686a4facd8f8b86c68a6df09"
server_key_iv: "7b832d5fb05e5a4a30312deacfb9a927"
server_key_version: "encryptionkey@0"
I20260926 07:22:09.532526 10955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:22:09.535125 10970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:09.536460 10955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:22:09.536571 10955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "f75c23e2671246f78a4fcbf88d48a502"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "9f331d7d686a4facd8f8b86c68a6df09"
server_key_iv: "7b832d5fb05e5a4a30312deacfb9a927"
server_key_version: "encryptionkey@0"
I20260926 07:22:09.536664 10955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:09.546461 10955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:09.549993 10955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:09.550148 10955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:09.550669 10955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:09.551461 10955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:09.551507 10955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:09.551548 10955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:09.551563 10955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:09.561616 10955 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:36133
I20260926 07:22:09.561646 11083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:36133 every 8 connection(s)
I20260926 07:22:09.562623 10955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:22:09.572122 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 10955
Sep 26 07:22:09 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:09.572291 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:22:09.572645 29615 external_mini_cluster.cc:1428] Setting key b519375742406586f2d29246428cf523
I20260926 07:22:09.577178 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:09.564494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:47979 (local address 127.28.235.254:42613)
0926 07:22:09.564671 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:09.564676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:09.565559 (+ 883us) server_negotiation.cc:407] Connection header received
0926 07:22:09.566863 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:09.566866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:09.566914 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:09.566988 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:09.568204 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:09.568785 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:09.570023 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:09.570262 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:09.570358 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:09.573063 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:09.573084 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:09.573090 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:09.573122 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:09.574909 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:09.575458 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:09.575465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:09.575470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:09.575569 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:09.575925 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:09.575930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:09.575932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:09.576133 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:09.576246 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:09.576783 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 07:22:09.576973 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:22:09.578249 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:09.564912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:47979)
0926 07:22:09.565397 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:09.565441 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:09.566654 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:09.567155 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:09.567162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:09.567545 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:09.568051 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:09.568064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:09.568908 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:09.568911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:09.569899 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:09.569908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:09.570422 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:09.570425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:09.570569 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:09.571146 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:09.571162 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:09.572788 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:09.575032 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:09.575037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:09.575052 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:09.575352 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:09.575667 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:09.575669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:09.575671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:09.575813 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:09.576245 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:09.576253 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:09.576540 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 07:22:09.576743 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 07:22:09.577124 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":112,"threads_started":1}
I20260926 07:22:09.580113 11084 heartbeater.cc:347] Connected to a master server at 127.28.235.254:42613
I20260926 07:22:09.580390 11084 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:09.580940 11084 heartbeater.cc:511] Master 127.28.235.254:42613 requested a full tablet report, sending...
I20260926 07:22:09.582775 10893 ts_manager.cc:194] Registered new tserver with Master: f75c23e2671246f78a4fcbf88d48a502 (127.28.235.193:36133)
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:22:09.584036 10893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:47979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:22:09.605336 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:42613
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:09.718981 11089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:09.719267 11089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:09.719312 11089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:09.723573 11089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:09.723661 11089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:09.723744 11089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:22:09.729262 11089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:09.730387 11089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:09.731401 11089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:09.741202 11095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:09.741211 11097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:09.741194 11094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:09.741570 11089 server_base.cc:1034] running on GCE node
I20260926 07:22:09.742025 11089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:09.742775 11089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:09.743976 11089 hybrid_clock.cc:648] HybridClock initialized: now 1790407329743962 us; error 27 us; skew 500 ppm
Sep 26 07:22:09 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:09.746984 11089 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:22:09.747989 11089 webserver.cc:460] Webserver started at http://127.28.235.194:36841/ using document root <none> and password file <none>
I20260926 07:22:09.748697 11089 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:09.748790 11089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:09.749064 11089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:09.751303 11089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "73aba1bfe3054207bba71f56fd9a18b7"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "6c32e80aa3c8f30ef604bc6bbbeb8371"
server_key_iv: "97451c5f254690d34561826cb82a4bfd"
server_key_version: "encryptionkey@0"
I20260926 07:22:09.751948 11089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "73aba1bfe3054207bba71f56fd9a18b7"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "6c32e80aa3c8f30ef604bc6bbbeb8371"
server_key_iv: "97451c5f254690d34561826cb82a4bfd"
server_key_version: "encryptionkey@0"
I20260926 07:22:09.756160 11089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:22:09.759105 11104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:09.760501 11089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:22:09.760725 11089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "73aba1bfe3054207bba71f56fd9a18b7"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "6c32e80aa3c8f30ef604bc6bbbeb8371"
server_key_iv: "97451c5f254690d34561826cb82a4bfd"
server_key_version: "encryptionkey@0"
I20260926 07:22:09.760872 11089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:09.787703 11089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:09.791139 11089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:09.791386 11089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:09.792053 11089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:09.792965 11089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:09.793069 11089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:09.793164 11089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:09.793218 11089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:09.802656 11089 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:40363
I20260926 07:22:09.802672 11217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:40363 every 8 connection(s)
I20260926 07:22:09.803622 11089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:22:09.812876 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 11089
I20260926 07:22:09.813009 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:22:09.813356 29615 external_mini_cluster.cc:1428] Setting key 4618c22089e2d924dc2e964191c1a95b
Sep 26 07:22:09 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:09.819196 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:09.805687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:55815 (local address 127.28.235.254:42613)
0926 07:22:09.805869 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:09.805874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:09.806834 (+ 960us) server_negotiation.cc:407] Connection header received
0926 07:22:09.808309 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:09.808312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:09.808365 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:09.808444 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:09.809810 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:09.810395 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:09.811618 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:09.811847 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:09.811934 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:09.814943 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:09.814947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:09.814949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:09.814996 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:09.816719 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:09.817417 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:09.817423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:09.817427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:09.817528 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:09.817876 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:09.817881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:09.817885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:09.818153 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:09.818260 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:09.818795 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 07:22:09.818945 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:22:09.819855 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:09.806087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:55815)
0926 07:22:09.806643 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:09.806691 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:09.808084 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:09.808614 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:09.808621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:09.808998 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:09.809599 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:09.809614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:09.810525 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:09.810528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:09.811491 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:09.811500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:09.812021 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:09.812024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:09.812178 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:09.812937 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:09.812961 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:09.814765 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:09.816876 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:09.816884 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:09.816905 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:09.817293 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:09.817628 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:09.817630 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:09.817631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:09.817772 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:09.818258 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:09.818263 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:09.818529 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 07:22:09.818683 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 07:22:09.818952 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":116,"threads_started":1}
I20260926 07:22:09.821200 11218 heartbeater.cc:347] Connected to a master server at 127.28.235.254:42613
I20260926 07:22:09.821504 11218 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:09.822132 11218 heartbeater.cc:511] Master 127.28.235.254:42613 requested a full tablet report, sending...
I20260926 07:22:09.823377 10893 ts_manager.cc:194] Registered new tserver with Master: 73aba1bfe3054207bba71f56fd9a18b7 (127.28.235.194:40363)
I20260926 07:22:09.823967 10893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:55815
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:22:09.846292 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:42613
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:09.965117 11223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:09.965481 11223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:09.965533 11223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:09.969677 11223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:09.969746 11223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:09.969807 11223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:22:09.975481 11223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:09.976632 11223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:09.977638 11223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:09.987784 11228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:09.987895 11223 server_base.cc:1034] running on GCE node
W20260926 07:22:09.987926 11229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:09.988096 11231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:09.988629 11223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:09.989432 11223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:09.990628 11223 hybrid_clock.cc:648] HybridClock initialized: now 1790407329990611 us; error 30 us; skew 500 ppm
Sep 26 07:22:09 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:09.993575 11223 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:22:09.994486 11223 webserver.cc:460] Webserver started at http://127.28.235.195:40631/ using document root <none> and password file <none>
I20260926 07:22:09.995066 11223 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:09.995162 11223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:09.995400 11223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:09.997644 11223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "b39868d60a2943e49b38bec1b65ff480"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "fc5882a3b22711067cd93ff4de84ba24"
server_key_iv: "9a2e00b7c468b62248ab8bc592b9df21"
server_key_version: "encryptionkey@0"
I20260926 07:22:09.998224 11223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "b39868d60a2943e49b38bec1b65ff480"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "fc5882a3b22711067cd93ff4de84ba24"
server_key_iv: "9a2e00b7c468b62248ab8bc592b9df21"
server_key_version: "encryptionkey@0"
I20260926 07:22:10.002170 11223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:22:10.004974 11238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:10.006402 11223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:10.006573 11223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "b39868d60a2943e49b38bec1b65ff480"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "fc5882a3b22711067cd93ff4de84ba24"
server_key_iv: "9a2e00b7c468b62248ab8bc592b9df21"
server_key_version: "encryptionkey@0"
I20260926 07:22:10.006716 11223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:10.030323 11223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:10.033809 11223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:10.034036 11223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:10.034749 11223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:10.035636 11223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:10.035710 11223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:10.035791 11223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:10.035845 11223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:10.045416 11223 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:46785
I20260926 07:22:10.045431 11351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:46785 every 8 connection(s)
I20260926 07:22:10.046413 11223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:22:10.054145 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 11223
I20260926 07:22:10.054255 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:22:10.054538 29615 external_mini_cluster.cc:1428] Setting key d672a889980d3b2c56f315def4ae900e
Sep 26 07:22:10 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407329, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:10.062095 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.048479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:50395 (local address 127.28.235.254:42613)
0926 07:22:10.048627 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:10.048631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:10.049687 (+ 1056us) server_negotiation.cc:407] Connection header received
0926 07:22:10.051183 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:10.051187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:10.051253 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:10.051343 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:10.052608 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.053259 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.054606 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.054922 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.055035 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.058067 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:10.058070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:10.058073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:10.058100 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:10.059823 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:10.060354 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:10.060357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:10.060359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:10.060455 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:10.060841 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:10.060845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:10.060846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:10.061190 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:10.061289 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:10.061841 (+ 552us) server_negotiation.cc:299] Negotiation successful
0926 07:22:10.061965 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:22:10.063021 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.048921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:50395)
0926 07:22:10.049523 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:10.049573 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:10.050948 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:10.051528 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:10.051536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:10.051941 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:10.052449 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.052463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.053415 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.053420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.054477 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.054490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.055156 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.055162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.055391 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.056203 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:10.056224 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:10.057889 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:10.059941 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:10.059946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:10.059960 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:10.060236 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:10.060565 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:10.060567 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:10.060569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:10.060709 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:10.061298 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:10.061304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:10.061592 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 07:22:10.061759 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 07:22:10.062039 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":139,"threads_started":1}
I20260926 07:22:10.064442 11352 heartbeater.cc:347] Connected to a master server at 127.28.235.254:42613
I20260926 07:22:10.064714 11352 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:10.065326 11352 heartbeater.cc:511] Master 127.28.235.254:42613 requested a full tablet report, sending...
I20260926 07:22:10.066583 10893 ts_manager.cc:194] Registered new tserver with Master: b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785)
I20260926 07:22:10.067265 10893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:50395
I20260926 07:22:10.068864 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:22:10.078747 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.070918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.28.235.254:42613)
0926 07:22:10.071045 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:10.071049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:10.071250 (+ 201us) server_negotiation.cc:407] Connection header received
0926 07:22:10.071338 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:10.071340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:10.071377 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:10.071443 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:10.072168 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.072668 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.073615 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.073874 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.073965 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.075322 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:10.075325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:10.075327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:10.075352 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:10.076922 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:10.077434 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:10.077436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:10.077438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:10.077514 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:10.077835 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:10.077837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:10.077839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:10.078060 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:10.078166 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:10.078438 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 07:22:10.078549 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:22:10.082074 10893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:22:10.085340 10893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:22:10.101408 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.094760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.28.235.194:40363)
0926 07:22:10.095464 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:10.095478 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:10.095860 (+ 382us) server_negotiation.cc:407] Connection header received
0926 07:22:10.096054 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:10.096058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:10.096228 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:10.096340 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:10.096748 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.097940 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.099859 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.100620 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.100730 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.100896 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:10.101112 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 07:22:10.101232 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":132,"threads_started":1}
I20260926 07:22:10.101433 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.094608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:36133 (local address 127.0.0.1:36894)
0926 07:22:10.095401 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:10.095451 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:10.095598 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:10.096384 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:10.096391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:10.096413 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:10.096554 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.096561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.098115 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.098117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.099511 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.099520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.100624 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.100627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.100773 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.100870 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 07:22:10.100992 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 07:22:10.101236 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":219,"thread_start_us":82,"threads_started":1}
I20260926 07:22:10.101433 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.094608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:40363 (local address 127.0.0.1:34146)
0926 07:22:10.095759 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:10.095773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:10.095924 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:10.096522 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:10.096525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:10.096539 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:10.096625 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.096631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.098069 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.098072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.099719 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.099731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.100732 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.100735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.100886 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.100905 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:22:10.100992 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:22:10.101241 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":938,"thread_start_us":89,"threads_started":1}
I20260926 07:22:10.102144 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.094864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.28.235.193:36133)
0926 07:22:10.095284 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:10.095291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:10.095746 (+ 455us) server_negotiation.cc:407] Connection header received
0926 07:22:10.095840 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:10.095846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:10.096061 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:10.096194 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:10.096683 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.097940 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.099640 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.100493 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.100634 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.100842 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:10.101664 (+ 822us) server_negotiation.cc:299] Negotiation successful
0926 07:22:10.101977 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":105,"threads_started":1}
I20260926 07:22:10.102705 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.095008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:46785 (local address 127.0.0.1:38786)
0926 07:22:10.095556 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:10.095570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:10.096172 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:10.096649 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:10.096652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:10.096669 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:10.096801 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.096810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.097895 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.097902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.099719 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.099731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.102310 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.102314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.102414 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.102437 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:22:10.102508 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:22:10.102581 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":187,"thread_start_us":104,"threads_started":1}
I20260926 07:22:10.103129 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.095505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.28.235.195:46785)
0926 07:22:10.095945 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:10.095959 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:10.095983 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:22:10.096340 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:10.096346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:10.096514 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:10.096670 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:10.096939 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.097747 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.100954 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.101986 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.102105 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.102240 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:10.102876 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 07:22:10.102980 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":147,"threads_started":1}
I20260926 07:22:10.104931 11152 tablet_service.cc:1467] Processing CreateTablet for tablet 266d5667336541759e4979abb3a22924 (DEFAULT_TABLE table=test-table [id=19951b137fb1493da4ab7be4baaf2273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:10.105460 11286 tablet_service.cc:1467] Processing CreateTablet for tablet 266d5667336541759e4979abb3a22924 (DEFAULT_TABLE table=test-table [id=19951b137fb1493da4ab7be4baaf2273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:10.106099 11152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266d5667336541759e4979abb3a22924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:10.106492 11286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266d5667336541759e4979abb3a22924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:10.107343 11018 tablet_service.cc:1467] Processing CreateTablet for tablet 266d5667336541759e4979abb3a22924 (DEFAULT_TABLE table=test-table [id=19951b137fb1493da4ab7be4baaf2273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:10.108353 11018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266d5667336541759e4979abb3a22924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:10.113453 11369 tablet_bootstrap.cc:492] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: Bootstrap starting.
I20260926 07:22:10.115489 11371 tablet_bootstrap.cc:492] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Bootstrap starting.
I20260926 07:22:10.116024 11370 tablet_bootstrap.cc:492] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: Bootstrap starting.
I20260926 07:22:10.117340 11369 tablet_bootstrap.cc:654] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:10.117909 11371 tablet_bootstrap.cc:654] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:10.118036 11370 tablet_bootstrap.cc:654] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:10.118285 11369 log.cc:824] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:10.118738 11371 log.cc:824] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:10.119254 11370 log.cc:824] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:10.120533 11369 tablet_bootstrap.cc:492] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: No bootstrap required, opened a new log
I20260926 07:22:10.120698 11369 ts_tablet_manager.cc:1397] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:22:10.120774 11371 tablet_bootstrap.cc:492] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: No bootstrap required, opened a new log
I20260926 07:22:10.121064 11371 ts_tablet_manager.cc:1397] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:22:10.121209 11370 tablet_bootstrap.cc:492] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: No bootstrap required, opened a new log
I20260926 07:22:10.121385 11370 ts_tablet_manager.cc:1397] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:22:10.123750 11369 raft_consensus.cc:348] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.124022 11369 raft_consensus.cc:374] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:10.124085 11369 raft_consensus.cc:729] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73aba1bfe3054207bba71f56fd9a18b7, State: Initialized, Role: FOLLOWER
I20260926 07:22:10.124604 11369 consensus_queue.cc:260] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.125293 11218 heartbeater.cc:503] Master 127.28.235.254:42613 was elected leader, sending a full tablet report...
I20260926 07:22:10.125206 11370 raft_consensus.cc:348] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.125461 11370 raft_consensus.cc:374] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:10.125525 11370 raft_consensus.cc:729] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39868d60a2943e49b38bec1b65ff480, State: Initialized, Role: FOLLOWER
I20260926 07:22:10.125967 11370 consensus_queue.cc:260] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.126263 11371 raft_consensus.cc:348] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.126513 11371 raft_consensus.cc:374] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:10.126559 11371 raft_consensus.cc:729] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75c23e2671246f78a4fcbf88d48a502, State: Initialized, Role: FOLLOWER
I20260926 07:22:10.126852 11369 ts_tablet_manager.cc:1428] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:22:10.127004 11370 ts_tablet_manager.cc:1428] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:22:10.126958 11371 consensus_queue.cc:260] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.127358 11352 heartbeater.cc:503] Master 127.28.235.254:42613 was elected leader, sending a full tablet report...
I20260926 07:22:10.127840 11371 ts_tablet_manager.cc:1428] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:22:10.128428 11084 heartbeater.cc:503] Master 127.28.235.254:42613 was elected leader, sending a full tablet report...
W20260926 07:22:10.297458 11353 tablet.cc:2369] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:22:10.304823 11219 tablet.cc:2369] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:22:10.314028 11085 tablet.cc:2369] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:22:10.331277 11375 raft_consensus.cc:482] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:22:10.331482 11375 raft_consensus.cc:504] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.332614 11375 leader_election.cc:288] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f75c23e2671246f78a4fcbf88d48a502 (127.28.235.193:36133), b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785)
I20260926 07:22:10.336910 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.332921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:46785 (local address 127.28.235.194:36231)
0926 07:22:10.333283 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:10.333305 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:10.333412 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:10.333704 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:10.333710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:10.333733 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:10.333886 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.333893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.334687 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.334690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.335705 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.335714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.336435 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.336447 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.336543 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.336574 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:22:10.336650 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 07:22:10.336736 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":91,"thread_start_us":122,"threads_started":1}
I20260926 07:22:10.336910 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.333055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:36231 (local address 127.28.235.195:46785)
0926 07:22:10.333206 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:10.333212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:10.333401 (+ 189us) server_negotiation.cc:407] Connection header received
0926 07:22:10.333520 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:10.333524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:10.333586 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:10.333669 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:10.334009 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.334564 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.335839 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.336296 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.336402 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.336495 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:10.336679 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 07:22:10.336774 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:22:10.336983 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.333120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:60595 (local address 127.28.235.193:36133)
0926 07:22:10.333241 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:10.333246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:10.333489 (+ 243us) server_negotiation.cc:407] Connection header received
0926 07:22:10.333603 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:10.333605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:10.333670 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:10.333748 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:10.334063 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.334565 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.335930 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.336340 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.336493 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.336549 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:10.336773 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 07:22:10.336859 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:22:10.336989 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.332961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:36133 (local address 127.28.235.194:60595)
0926 07:22:10.333355 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:10.333392 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:10.333507 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:10.333785 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:10.333788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:10.333806 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:10.333926 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.333931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.334683 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.334689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.335792 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.335808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.336495 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.336499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.336610 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.336660 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 07:22:10.336747 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:22:10.336828 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":79,"thread_start_us":122,"threads_started":1}
I20260926 07:22:10.337595 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266d5667336541759e4979abb3a22924" candidate_uuid: "73aba1bfe3054207bba71f56fd9a18b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75c23e2671246f78a4fcbf88d48a502" is_pre_election: true
I20260926 07:22:10.337607 11306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266d5667336541759e4979abb3a22924" candidate_uuid: "73aba1bfe3054207bba71f56fd9a18b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39868d60a2943e49b38bec1b65ff480" is_pre_election: true
I20260926 07:22:10.337914 11038 raft_consensus.cc:2454] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73aba1bfe3054207bba71f56fd9a18b7 in term 0.
I20260926 07:22:10.337973 11306 raft_consensus.cc:2454] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73aba1bfe3054207bba71f56fd9a18b7 in term 0.
I20260926 07:22:10.338488 11108 leader_election.cc:302] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 73aba1bfe3054207bba71f56fd9a18b7, b39868d60a2943e49b38bec1b65ff480, f75c23e2671246f78a4fcbf88d48a502; no voters:
I20260926 07:22:10.338815 11375 raft_consensus.cc:2781] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:22:10.338918 11375 raft_consensus.cc:482] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:22:10.338979 11375 raft_consensus.cc:3037] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:10.340031 11375 raft_consensus.cc:504] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.340705 11375 leader_election.cc:288] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [CANDIDATE]: Term 1 election: Requested vote from peers f75c23e2671246f78a4fcbf88d48a502 (127.28.235.193:36133), b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785)
I20260926 07:22:10.340870 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266d5667336541759e4979abb3a22924" candidate_uuid: "73aba1bfe3054207bba71f56fd9a18b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75c23e2671246f78a4fcbf88d48a502"
I20260926 07:22:10.340886 11306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266d5667336541759e4979abb3a22924" candidate_uuid: "73aba1bfe3054207bba71f56fd9a18b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39868d60a2943e49b38bec1b65ff480"
I20260926 07:22:10.340992 11038 raft_consensus.cc:3037] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:10.341102 11306 raft_consensus.cc:3037] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:10.342154 11038 raft_consensus.cc:2454] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73aba1bfe3054207bba71f56fd9a18b7 in term 1.
I20260926 07:22:10.342324 11306 raft_consensus.cc:2454] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73aba1bfe3054207bba71f56fd9a18b7 in term 1.
I20260926 07:22:10.342491 11105 leader_election.cc:302] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73aba1bfe3054207bba71f56fd9a18b7, f75c23e2671246f78a4fcbf88d48a502; no voters:
I20260926 07:22:10.342676 11375 raft_consensus.cc:2781] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:22:10.342927 11375 raft_consensus.cc:686] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 LEADER]: Becoming Leader. State: Replica: 73aba1bfe3054207bba71f56fd9a18b7, State: Running, Role: LEADER
I20260926 07:22:10.343267 11375 consensus_queue.cc:237] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.346637 10892 catalog_manager.cc:5495] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73aba1bfe3054207bba71f56fd9a18b7 (127.28.235.194). New cstate: current_term: 1 leader_uuid: "73aba1bfe3054207bba71f56fd9a18b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } health_report { overall_health: UNKNOWN } } }
I20260926 07:22:10.394860 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.391082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.28.235.194:40363)
0926 07:22:10.391229 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:10.391234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:10.391249 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:22:10.391412 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:10.391414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:10.391474 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:10.391605 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:22:10.391855 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.392378 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.393348 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.393565 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.393675 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.393990 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:22:10.394425 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:22:10.394564 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:10.394648 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 07:22:10.394709 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:22:10.401168 11306 raft_consensus.cc:1261] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480 [term 1 FOLLOWER]: Refusing update from remote peer 73aba1bfe3054207bba71f56fd9a18b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:10.401187 11038 raft_consensus.cc:1261] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Refusing update from remote peer 73aba1bfe3054207bba71f56fd9a18b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:10.401949 11375 consensus_queue.cc:1035] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:10.402163 11380 consensus_queue.cc:1035] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:10.412397 11384 mvcc.cc:205] Tried to move back new op lower bound from 7333508425315500032 to 7333508425088868352. Current Snapshot: MvccSnapshot[applied={T|T < 7333508425315500032}]
I20260926 07:22:10.415625 11386 mvcc.cc:205] Tried to move back new op lower bound from 7333508425315500032 to 7333508425088868352. Current Snapshot: MvccSnapshot[applied={T|T < 7333508425315500032 or (T in {7333508425315500032})}]
I20260926 07:22:10.420691 10892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:22:10.420845 10892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:22:10.423395 10892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Sending DeleteTablet for 3 replicas of tablet 266d5667336541759e4979abb3a22924
I20260926 07:22:10.424202 11152 tablet_service.cc:1514] Processing DeleteTablet for tablet 266d5667336541759e4979abb3a22924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:10 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:34146
I20260926 07:22:10.424435 11286 tablet_service.cc:1514] Processing DeleteTablet for tablet 266d5667336541759e4979abb3a22924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:10 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:38786
I20260926 07:22:10.424660 11018 tablet_service.cc:1514] Processing DeleteTablet for tablet 266d5667336541759e4979abb3a22924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:10 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:36894
I20260926 07:22:10.424787 11393 tablet_replica.cc:330] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: stopping tablet replica
I20260926 07:22:10.425160 11393 raft_consensus.cc:2229] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:22:10.425266 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 10955
I20260926 07:22:10.426486 11393 raft_consensus.cc:2258] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:10.428220 11393 ts_tablet_manager.cc:1905] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:10.429286 11394 tablet_replica.cc:330] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: stopping tablet replica
I20260926 07:22:10.429723 11394 raft_consensus.cc:2229] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:22:10.430147 11394 raft_consensus.cc:2258] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:10.431859 11394 ts_tablet_manager.cc:1905] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:10.431998 11393 ts_tablet_manager.cc:1918] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:22:10.432121 11393 log.cc:1196] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/266d5667336541759e4979abb3a22924
I20260926 07:22:10.432464 11393 ts_tablet_manager.cc:1939] T 266d5667336541759e4979abb3a22924 P 73aba1bfe3054207bba71f56fd9a18b7: Deleting consensus metadata
I20260926 07:22:10.433861 10879 catalog_manager.cc:4841] TS 73aba1bfe3054207bba71f56fd9a18b7 (127.28.235.194:40363): tablet 266d5667336541759e4979abb3a22924 (table test-table [id=19951b137fb1493da4ab7be4baaf2273]) successfully deleted
W20260926 07:22:10.434000 10877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:36133 (error 108)
I20260926 07:22:10.434871 11394 ts_tablet_manager.cc:1918] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:22:10.434967 11394 log.cc:1196] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/266d5667336541759e4979abb3a22924
I20260926 07:22:10.435035 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.434717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:36133 (local address 127.0.0.1:36900)
0926 07:22:10.434851 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:10.434928 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:36133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 07:22:10.435159 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 11089
I20260926 07:22:10.435276 11394 ts_tablet_manager.cc:1939] T 266d5667336541759e4979abb3a22924 P b39868d60a2943e49b38bec1b65ff480: Deleting consensus metadata
W20260926 07:22:10.436638 10877 catalog_manager.cc:4584] TS f75c23e2671246f78a4fcbf88d48a502 (127.28.235.193:36133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 266d5667336541759e4979abb3a22924: Network error: Client connection negotiation failed: client connection to 127.28.235.193:36133: connect: Connection refused (error 111)
I20260926 07:22:10.437105 10880 catalog_manager.cc:4841] TS b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785): tablet 266d5667336541759e4979abb3a22924 (table test-table [id=19951b137fb1493da4ab7be4baaf2273]) successfully deleted
I20260926 07:22:10.441746 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 11223
I20260926 07:22:10.447384 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 10861
I20260926 07:22:10.453718 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:42613
--webserver_interface=127.28.235.254
--webserver_port=38725
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:42613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:10.566334 11397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:10.566627 11397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:10.566668 11397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:10.570837 11397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:22:10.570902 11397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:10.570923 11397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:10.570940 11397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:22:10.570956 11397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:22:10.576609 11397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:42613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:42613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=38725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:10.577783 11397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:10.578801 11397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:10.593115 11406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:10.593086 11403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:10.593176 11397 server_base.cc:1034] running on GCE node
W20260926 07:22:10.593086 11404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:10.593940 11397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:10.595157 11397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:10.596333 11397 hybrid_clock.cc:648] HybridClock initialized: now 1790407330596297 us; error 53 us; skew 500 ppm
Sep 26 07:22:10 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:10.599282 11397 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:22:10.600294 11397 webserver.cc:460] Webserver started at http://127.28.235.254:38725/ using document root <none> and password file <none>
I20260926 07:22:10.600937 11397 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:10.601064 11397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:10.605461 11397 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:22:10.608352 11413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:10.610951 11397 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:22:10.611114 11397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "0cf06e20743d46ea86f0f7e477217cf3"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "c93217f68301030105b47b087a031ebb"
server_key_iv: "82122f06b6e21b5eea03ec9074d44897"
server_key_version: "encryptionkey@0"
I20260926 07:22:10.611783 11397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:10.642223 11397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:10.645740 11397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:10.645967 11397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:10.653595 11397 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:42613
I20260926 07:22:10.653908 11465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:42613 every 8 connection(s)
I20260926 07:22:10.654764 11397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:22:10.659713 11466 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:22:10.660751 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 11397
I20260926 07:22:10.661276 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:36133
--local_ip_for_outbound_sockets=127.28.235.193
--tserver_master_addrs=127.28.235.254:42613
--webserver_port=33015
--webserver_interface=127.28.235.193
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:22:10.661873 11466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Bootstrap starting.
I20260926 07:22:10.679980 11466 log.cc:824] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:10.692394 11466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:22:10.692803 11466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Bootstrap complete.
I20260926 07:22:10.697144 11466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:10.697543 11466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf06e20743d46ea86f0f7e477217cf3, State: Initialized, Role: FOLLOWER
I20260926 07:22:10.698048 11466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:10.698191 11466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:22:10.698263 11466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:22:10.698380 11466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:22:10.700335 11466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:10.700677 11466 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf06e20743d46ea86f0f7e477217cf3; no voters:
I20260926 07:22:10.701012 11466 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:22:10.701148 11470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:22:10.701356 11470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 2 LEADER]: Becoming Leader. State: Replica: 0cf06e20743d46ea86f0f7e477217cf3, State: Running, Role: LEADER
I20260926 07:22:10.701658 11470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:10.702306 11466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:22:10.703241 11471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0cf06e20743d46ea86f0f7e477217cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } } }
I20260926 07:22:10.703399 11471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:10.703245 11472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf06e20743d46ea86f0f7e477217cf3. Latest consensus state: current_term: 2 leader_uuid: "0cf06e20743d46ea86f0f7e477217cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } } }
I20260926 07:22:10.703629 11472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:10.707613 11486 catalog_manager.cc:1259] Loaded cluster ID: b13c0a8bf98f445583a863a4141052b9
I20260926 07:22:10.707715 11486 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: loading cluster ID for follower catalog manager: success
W20260926 07:22:10.708575 11486 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260926 07:22:10.708665 11486 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260926 07:22:10.708832 11474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:22:10.710983 11474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:22:10.711407 11474 catalog_manager.cc:1259] Loaded cluster ID: b13c0a8bf98f445583a863a4141052b9
I20260926 07:22:10.711477 11474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:22:10.712309 11474 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 07:22:10.715207 11474 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 07:22:10.720315 11474 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:22:10.721364 11474 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Loaded TSK: 0
I20260926 07:22:10.722481 11474 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:22:10.817190 11468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:10.817515 11468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:10.817618 11468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:10.821826 11468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:10.821936 11468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:10.822036 11468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:22:10.827742 11468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:36133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=33015
--tserver_master_addrs=127.28.235.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:10.828979 11468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:10.830133 11468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:10.841217 11493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:10.841223 11494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:10.841308 11496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:10.841676 11468 server_base.cc:1034] running on GCE node
I20260926 07:22:10.842186 11468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:10.843199 11468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:10.844413 11468 hybrid_clock.cc:648] HybridClock initialized: now 1790407330844376 us; error 48 us; skew 500 ppm
Sep 26 07:22:10 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:10.848080 11468 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:22:10.849274 11468 webserver.cc:460] Webserver started at http://127.28.235.193:33015/ using document root <none> and password file <none>
I20260926 07:22:10.850045 11468 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:10.850172 11468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:10.855758 11468 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 07:22:10.858523 11503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:10.859992 11468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:22:10.860140 11468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "f75c23e2671246f78a4fcbf88d48a502"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "9f331d7d686a4facd8f8b86c68a6df09"
server_key_iv: "7b832d5fb05e5a4a30312deacfb9a927"
server_key_version: "encryptionkey@0"
I20260926 07:22:10.860762 11468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:10.884471 11468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:10.888011 11468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:10.888221 11468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:10.888949 11468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:10.890197 11510 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:22:10.894482 11468 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:22:10.894554 11468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 07:22:10.894631 11468 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:22:10.896373 11468 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:22:10.896433 11468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 07:22:10.896688 11510 tablet_bootstrap.cc:492] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Bootstrap starting.
I20260926 07:22:10.907994 11468 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:36133
I20260926 07:22:10.908041 11617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:36133 every 8 connection(s)
I20260926 07:22:10.909370 11468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:22:10.918119 11510 log.cc:824] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:10.919489 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 11468
I20260926 07:22:10.919902 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:40363
--local_ip_for_outbound_sockets=127.28.235.194
--tserver_master_addrs=127.28.235.254:42613
--webserver_port=36841
--webserver_interface=127.28.235.194
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:22:10 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:10.927897 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.911703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:47343)
0926 07:22:10.912236 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:10.912278 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:10.913200 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:10.914335 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:10.914343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:10.914775 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:10.915365 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.915381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.916855 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.916858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.917827 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.917838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.918669 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.918672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.918823 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.919496 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:10.919518 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:10.921188 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:10.924055 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:10.924061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:10.924073 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:10.924451 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:10.924945 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:10.924949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:10.924951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:10.925177 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:10.926030 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:10.926036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:10.926349 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 07:22:10.926553 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 07:22:10.926903 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":166,"threads_started":1}
I20260926 07:22:10.928017 11510 tablet_bootstrap.cc:492] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:22:10.928476 11510 tablet_bootstrap.cc:492] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Bootstrap complete.
I20260926 07:22:10.928622 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.911657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:47343 (local address 127.28.235.254:42613)
0926 07:22:10.912055 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:10.912075 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:10.912402 (+ 327us) server_negotiation.cc:407] Connection header received
0926 07:22:10.913631 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:10.913641 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:10.913942 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:10.914272 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:10.915545 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.916706 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.917967 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.918502 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.918699 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.921438 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:10.921447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:10.921467 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:10.921509 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:10.923915 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:10.924614 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:10.924622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:10.924632 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:10.924822 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:10.925305 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:10.925312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:10.925316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:10.925883 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:10.926042 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:10.926751 (+ 709us) server_negotiation.cc:299] Negotiation successful
0926 07:22:10.927109 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":120,"threads_started":1}
I20260926 07:22:10.929904 11510 ts_tablet_manager.cc:1397] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.011s
I20260926 07:22:10.931324 11618 heartbeater.cc:347] Connected to a master server at 127.28.235.254:42613
I20260926 07:22:10.931595 11618 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:10.932236 11618 heartbeater.cc:511] Master 127.28.235.254:42613 requested a full tablet report, sending...
I20260926 07:22:10.932753 11510 raft_consensus.cc:348] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.933307 11510 raft_consensus.cc:729] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75c23e2671246f78a4fcbf88d48a502, State: Initialized, Role: FOLLOWER
I20260926 07:22:10.933828 11510 consensus_queue.cc:260] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } }
I20260926 07:22:10.934857 11510 ts_tablet_manager.cc:1428] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 07:22:10.935232 11430 ts_manager.cc:194] Registered new tserver with Master: f75c23e2671246f78a4fcbf88d48a502 (127.28.235.193:36133)
I20260926 07:22:10.939730 11430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:47343
I20260926 07:22:10.942682 11618 heartbeater.cc:503] Master 127.28.235.254:42613 was elected leader, sending a full tablet report...
Sep 26 07:22:10 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407330, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:22:10.950189 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.938670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:36133 (local address 127.0.0.1:36912)
0926 07:22:10.939115 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:10.939160 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:10.939289 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:10.940005 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:10.940011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:10.940363 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:10.940494 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.940506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.941987 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.941992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.943046 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:10.943059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.943791 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.943796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:10.943921 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.944606 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:10.944618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:10.946231 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:10.948379 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:10.948387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:10.948392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:10.948629 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:10.948952 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:10.948954 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:10.948956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:10.949092 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:10.949639 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:10.949644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:10.949754 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 07:22:10.949884 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 07:22:10.950051 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":161,"threads_started":1}
I20260926 07:22:10.950325 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:10.938732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.28.235.193:36133)
0926 07:22:10.939540 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:10.939549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:10.939567 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:22:10.939625 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:10.939629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:10.939789 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:10.939942 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:10.940665 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.941854 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.943176 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:10.943660 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:10.943785 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:10.946379 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:10.946390 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:10.946404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:10.946432 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:10.948249 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:10.948745 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:10.948751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:10.948753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:10.948847 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:10.949183 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:10.949187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:10.949188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:10.949518 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:10.949650 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:10.949988 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 07:22:10.950164 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":715,"thread_start_us":99,"threads_started":1}
I20260926 07:22:10.951350 11552 tablet_service.cc:1514] Processing DeleteTablet for tablet 266d5667336541759e4979abb3a22924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:10 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:36912
I20260926 07:22:10.952219 11627 tablet_replica.cc:330] stopping tablet replica
I20260926 07:22:10.952533 11627 raft_consensus.cc:2229] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:22:10.952791 11627 raft_consensus.cc:2258] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:10.953795 11627 ts_tablet_manager.cc:1905] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:10.957432 11627 ts_tablet_manager.cc:1918] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:22:10.957518 11627 log.cc:1196] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/266d5667336541759e4979abb3a22924
I20260926 07:22:10.957782 11627 ts_tablet_manager.cc:1939] T 266d5667336541759e4979abb3a22924 P f75c23e2671246f78a4fcbf88d48a502: Deleting consensus metadata
I20260926 07:22:10.959836 11414 catalog_manager.cc:4841] TS f75c23e2671246f78a4fcbf88d48a502 (127.28.235.193:36133): tablet 266d5667336541759e4979abb3a22924 (table test-table [id=19951b137fb1493da4ab7be4baaf2273]) successfully deleted
W20260926 07:22:11.045068 11623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:11.045449 11623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:11.045562 11623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:11.049614 11623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:11.049706 11623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:11.049796 11623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:22:11.055251 11623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:40363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=36841
--tserver_master_addrs=127.28.235.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:11.056394 11623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:11.057462 11623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:11.067852 11632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:11.067878 11635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:11.067859 11633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:11.068137 11623 server_base.cc:1034] running on GCE node
I20260926 07:22:11.068758 11623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:11.069552 11623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:11.070868 11623 hybrid_clock.cc:648] HybridClock initialized: now 1790407331070802 us; error 56 us; skew 500 ppm
Sep 26 07:22:11 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407331, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:11.074121 11623 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:22:11.075147 11623 webserver.cc:460] Webserver started at http://127.28.235.194:36841/ using document root <none> and password file <none>
I20260926 07:22:11.075747 11623 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:11.075824 11623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:11.080228 11623 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:22:11.082556 11642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:11.083931 11623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:22:11.084096 11623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "73aba1bfe3054207bba71f56fd9a18b7"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "6c32e80aa3c8f30ef604bc6bbbeb8371"
server_key_iv: "97451c5f254690d34561826cb82a4bfd"
server_key_version: "encryptionkey@0"
I20260926 07:22:11.084697 11623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:11.097170 11623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:11.100229 11623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:11.100448 11623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:11.101135 11623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:11.102049 11623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:11.102118 11623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:11.102177 11623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:11.102231 11623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:11.112505 11623 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:40363
I20260926 07:22:11.112545 11755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:40363 every 8 connection(s)
I20260926 07:22:11.113662 11623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:22:11.122172 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 11623
I20260926 07:22:11.122660 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:46785
--local_ip_for_outbound_sockets=127.28.235.195
--tserver_master_addrs=127.28.235.254:42613
--webserver_port=40631
--webserver_interface=127.28.235.195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:22:11 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407331, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:11.128881 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.115993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:60229 (local address 127.28.235.254:42613)
0926 07:22:11.116136 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:11.116139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:11.117078 (+ 939us) server_negotiation.cc:407] Connection header received
0926 07:22:11.118526 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:11.118530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:11.118594 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:11.118698 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:11.119963 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.120496 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.121785 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.122029 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.122147 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.124903 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:11.124909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:11.124913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:11.124948 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:11.126686 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:11.127272 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:11.127276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:11.127278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:11.127364 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:11.127721 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:11.127725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:11.127726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:11.127919 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:11.128032 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:11.128547 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 07:22:11.128692 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:22:11.129782 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.116370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:60229)
0926 07:22:11.116898 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:11.116955 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:11.118289 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:11.118899 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:11.118907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:11.119295 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:11.119805 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.119817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.120634 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.120637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.121649 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.121658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.122192 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.122195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.122425 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.123152 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:11.123169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:11.124703 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:11.126840 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:11.126846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:11.126863 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:11.127153 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:11.127462 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:11.127465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:11.127467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:11.127614 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:11.128029 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:11.128037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:11.128307 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 07:22:11.128507 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 07:22:11.128853 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
I20260926 07:22:11.131274 11756 heartbeater.cc:347] Connected to a master server at 127.28.235.254:42613
I20260926 07:22:11.131542 11756 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:11.132068 11756 heartbeater.cc:511] Master 127.28.235.254:42613 requested a full tablet report, sending...
I20260926 07:22:11.133303 11430 ts_manager.cc:194] Registered new tserver with Master: 73aba1bfe3054207bba71f56fd9a18b7 (127.28.235.194:40363)
I20260926 07:22:11.134223 11430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:60229
W20260926 07:22:11.241212 11759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:11.241492 11759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:11.241536 11759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:11.245636 11759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:11.245709 11759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:11.245769 11759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:22:11.251343 11759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:46785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=40631
--tserver_master_addrs=127.28.235.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:11.252462 11759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:11.253580 11759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:11.263942 11765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:11.263998 11759 server_base.cc:1034] running on GCE node
W20260926 07:22:11.263942 11764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:11.264289 11767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:11.264827 11759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:11.265585 11759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:11.266795 11759 hybrid_clock.cc:648] HybridClock initialized: now 1790407331266788 us; error 49 us; skew 500 ppm
Sep 26 07:22:11 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407331, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:11.270284 11759 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:22:11.271427 11759 webserver.cc:460] Webserver started at http://127.28.235.195:40631/ using document root <none> and password file <none>
I20260926 07:22:11.272084 11759 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:11.272176 11759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:11.276769 11759 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:11.279119 11774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:11.280395 11759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:22:11.280539 11759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "b39868d60a2943e49b38bec1b65ff480"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "fc5882a3b22711067cd93ff4de84ba24"
server_key_iv: "9a2e00b7c468b62248ab8bc592b9df21"
server_key_version: "encryptionkey@0"
I20260926 07:22:11.281184 11759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:11.314880 11759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:11.318787 11759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:11.319017 11759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:11.319684 11759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:11.320565 11759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:11.320621 11759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:11.320724 11759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:11.320780 11759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:11.330844 11759 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:46785
I20260926 07:22:11.330864 11887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:46785 every 8 connection(s)
I20260926 07:22:11.331981 11759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:22:11.341465 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 11759
Sep 26 07:22:11 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407331, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:11.348416 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.334239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:49605 (local address 127.28.235.254:42613)
0926 07:22:11.334397 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:11.334401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:11.335379 (+ 978us) server_negotiation.cc:407] Connection header received
0926 07:22:11.336948 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:11.336953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:11.337053 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:11.337178 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:11.338533 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.339134 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.340443 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.340714 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.340849 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.344207 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:11.344211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:11.344213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:11.344237 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:11.346028 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:11.346674 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:11.346679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:11.346681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:11.346767 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:11.347133 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:11.347138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:11.347140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:11.347316 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:11.347427 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:11.348108 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 07:22:11.348247 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:22:11.349551 11890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.334649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:49605)
0926 07:22:11.335190 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:11.335238 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:11.336694 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:11.337386 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:11.337395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:11.337808 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:11.338357 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.338372 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.339285 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.339297 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.340293 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.340304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.340897 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.340901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.341179 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.341931 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:11.341949 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:11.344023 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:11.346153 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:11.346161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:11.346181 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:11.346565 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:11.346877 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:11.346880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:11.346882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:11.347031 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:11.347436 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:11.347446 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:11.347798 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 07:22:11.348047 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 07:22:11.348414 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":128,"threads_started":1}
I20260926 07:22:11.351017 11888 heartbeater.cc:347] Connected to a master server at 127.28.235.254:42613
I20260926 07:22:11.351320 11888 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:11.351336 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.342240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.28.235.254:42613)
0926 07:22:11.342533 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:11.342537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:11.342752 (+ 215us) server_negotiation.cc:407] Connection header received
0926 07:22:11.342898 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:11.342904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:11.342972 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:11.343077 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:11.343966 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.344543 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.345916 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.346217 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.346334 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.347621 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:11.347624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:11.347626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:11.347649 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:11.349362 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:11.349891 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:11.349895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:11.349896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:11.349987 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:11.350374 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:11.350377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:11.350379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:11.350620 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:11.350830 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:11.351020 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 07:22:11.351167 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":92,"threads_started":1}
I20260926 07:22:11.351969 11888 heartbeater.cc:511] Master 127.28.235.254:42613 requested a full tablet report, sending...
I20260926 07:22:11.353454 11430 ts_manager.cc:194] Registered new tserver with Master: b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785)
I20260926 07:22:11.354115 11430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:49605
I20260926 07:22:11.354569 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:22:11.359216 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.355353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.28.235.254:42613)
0926 07:22:11.355466 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:11.355471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:11.355578 (+ 107us) server_negotiation.cc:407] Connection header received
0926 07:22:11.355770 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:11.355773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:11.355878 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:11.355986 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:22:11.356240 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.356765 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.357759 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.357999 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.358117 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.358444 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:22:11.358828 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:22:11.358961 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:11.359005 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 07:22:11.359071 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:22:11.359939 11430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:22:11.363423 11430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:22:11.372149 11552 tablet_service.cc:1467] Processing CreateTablet for tablet e86e2f071c8944ad8e2668e28265be6c (DEFAULT_TABLE table=test-table [id=13edce5eaed84b10a354214005c06024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:11.372964 11552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86e2f071c8944ad8e2668e28265be6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:11.376314 11510 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Bootstrap starting.
I20260926 07:22:11.376679 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.370932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:40363 (local address 127.0.0.1:34168)
0926 07:22:11.371257 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:11.371273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:11.371366 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:11.372262 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:11.372265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:11.372287 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:11.372414 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.372422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.373848 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.373851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.374736 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.374745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.376180 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.376184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.376305 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.376380 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 07:22:11.376442 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 07:22:11.376538 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260926 07:22:11.376951 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.371394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:46785 (local address 127.0.0.1:38794)
0926 07:22:11.371880 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:11.371893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:11.371973 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:11.372667 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:11.372671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:11.372690 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:11.372783 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.372789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.374592 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.374595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.375403 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.375411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.376550 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.376554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.376642 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.376664 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:22:11.376726 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 07:22:11.376817 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":110,"threads_started":1}
I20260926 07:22:11.377236 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.371833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.28.235.195:46785)
0926 07:22:11.372223 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:11.372235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:11.372256 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:22:11.372331 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:11.372337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:11.372514 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:11.372664 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:11.373103 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.374441 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.375531 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.376451 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.376567 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.376705 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:11.376862 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:22:11.376963 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":170,"threads_started":1}
I20260926 07:22:11.377442 11510 tablet_bootstrap.cc:654] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:11.378085 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.371304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.28.235.194:40363)
0926 07:22:11.371723 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:11.371738 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:11.371764 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:22:11.371852 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:11.371858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:11.372107 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:11.372271 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:11.372545 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.373712 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.374883 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.376016 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.377140 (+ 1124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.377287 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:11.377348 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 07:22:11.377975 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":192,"threads_started":1}
I20260926 07:22:11.378942 11510 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: No bootstrap required, opened a new log
I20260926 07:22:11.379038 11510 ts_tablet_manager.cc:1397] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:22:11.379488 11510 raft_consensus.cc:348] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:11.379619 11510 raft_consensus.cc:374] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:11.379658 11510 raft_consensus.cc:729] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75c23e2671246f78a4fcbf88d48a502, State: Initialized, Role: FOLLOWER
I20260926 07:22:11.379781 11510 consensus_queue.cc:260] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:11.380056 11510 ts_tablet_manager.cc:1428] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:22:11.380654 11822 tablet_service.cc:1467] Processing CreateTablet for tablet e86e2f071c8944ad8e2668e28265be6c (DEFAULT_TABLE table=test-table [id=13edce5eaed84b10a354214005c06024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:11.381997 11822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86e2f071c8944ad8e2668e28265be6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:11.383591 11690 tablet_service.cc:1467] Processing CreateTablet for tablet e86e2f071c8944ad8e2668e28265be6c (DEFAULT_TABLE table=test-table [id=13edce5eaed84b10a354214005c06024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:11.384721 11690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86e2f071c8944ad8e2668e28265be6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:11.389489 11901 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Bootstrap starting.
I20260926 07:22:11.392517 11901 tablet_bootstrap.cc:654] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:11.392583 11902 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Bootstrap starting.
I20260926 07:22:11.393483 11901 log.cc:824] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:11.394969 11902 tablet_bootstrap.cc:654] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:11.395547 11901 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: No bootstrap required, opened a new log
I20260926 07:22:11.395779 11901 ts_tablet_manager.cc:1397] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:22:11.395857 11902 log.cc:824] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:11.397835 11902 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: No bootstrap required, opened a new log
I20260926 07:22:11.398036 11902 ts_tablet_manager.cc:1397] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:22:11.399287 11901 raft_consensus.cc:348] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:11.399542 11901 raft_consensus.cc:374] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:11.399701 11901 raft_consensus.cc:729] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39868d60a2943e49b38bec1b65ff480, State: Initialized, Role: FOLLOWER
I20260926 07:22:11.400179 11901 consensus_queue.cc:260] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:11.400898 11888 heartbeater.cc:503] Master 127.28.235.254:42613 was elected leader, sending a full tablet report...
I20260926 07:22:11.401207 11901 ts_tablet_manager.cc:1428] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260926 07:22:11.401202 11902 raft_consensus.cc:348] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:11.401479 11902 raft_consensus.cc:374] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:11.401535 11902 raft_consensus.cc:729] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73aba1bfe3054207bba71f56fd9a18b7, State: Initialized, Role: FOLLOWER
I20260926 07:22:11.402007 11902 consensus_queue.cc:260] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:11.403131 11756 heartbeater.cc:503] Master 127.28.235.254:42613 was elected leader, sending a full tablet report...
I20260926 07:22:11.403169 11902 ts_tablet_manager.cc:1428] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260926 07:22:11.410815 11619 tablet.cc:2369] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:22:11.583077 11889 tablet.cc:2369] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:22:11.615233 11757 tablet.cc:2369] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:22:11.802917 11905 raft_consensus.cc:482] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:22:11.803170 11905 raft_consensus.cc:504] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:11.804538 11905 leader_election.cc:288] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73aba1bfe3054207bba71f56fd9a18b7 (127.28.235.194:40363), f75c23e2671246f78a4fcbf88d48a502 (127.28.235.193:36133)
I20260926 07:22:11.808519 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.805090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:59933 (local address 127.28.235.193:36133)
0926 07:22:11.805427 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:11.805430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:11.805441 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:22:11.805488 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:11.805490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:11.805542 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:11.805666 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:11.806001 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.806523 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.807533 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.807973 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.808108 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.808287 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:11.808326 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 07:22:11.808394 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
I20260926 07:22:11.808516 11890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.804937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:36133 (local address 127.28.235.195:59933)
0926 07:22:11.805107 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:11.805123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:11.805235 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:11.805676 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:11.805679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:11.805754 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:11.805863 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.805868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.806641 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.806645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.807407 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.807416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.808108 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.808112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.808212 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.808234 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:22:11.808295 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 07:22:11.808373 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:22:11.808645 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.805065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:54689 (local address 127.28.235.194:40363)
0926 07:22:11.805196 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:11.805203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:11.805498 (+ 295us) server_negotiation.cc:407] Connection header received
0926 07:22:11.805706 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:11.805709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:11.805752 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:11.805839 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:11.806136 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.806654 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.807683 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.808065 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.808189 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.808245 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:11.808434 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:22:11.808501 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:22:11.808699 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.804937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:40363 (local address 127.28.235.195:54689)
0926 07:22:11.805394 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:11.805405 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:11.805534 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:11.805883 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:11.805886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:11.805903 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:11.806025 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.806030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.806766 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.806772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.807557 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:11.807565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.808184 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.808188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:11.808303 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.808323 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:22:11.808418 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 07:22:11.808489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":143,"thread_start_us":83,"threads_started":1}
I20260926 07:22:11.809388 11710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86e2f071c8944ad8e2668e28265be6c" candidate_uuid: "b39868d60a2943e49b38bec1b65ff480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73aba1bfe3054207bba71f56fd9a18b7" is_pre_election: true
I20260926 07:22:11.809366 11572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86e2f071c8944ad8e2668e28265be6c" candidate_uuid: "b39868d60a2943e49b38bec1b65ff480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75c23e2671246f78a4fcbf88d48a502" is_pre_election: true
I20260926 07:22:11.809672 11572 raft_consensus.cc:2454] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b39868d60a2943e49b38bec1b65ff480 in term 0.
I20260926 07:22:11.809685 11710 raft_consensus.cc:2454] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b39868d60a2943e49b38bec1b65ff480 in term 0.
I20260926 07:22:11.810168 11777 leader_election.cc:302] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 73aba1bfe3054207bba71f56fd9a18b7, b39868d60a2943e49b38bec1b65ff480, f75c23e2671246f78a4fcbf88d48a502; no voters:
I20260926 07:22:11.810513 11905 raft_consensus.cc:2781] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:22:11.810599 11905 raft_consensus.cc:482] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:22:11.810642 11905 raft_consensus.cc:3037] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:11.811848 11905 raft_consensus.cc:504] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:11.812214 11905 leader_election.cc:288] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [CANDIDATE]: Term 1 election: Requested vote from peers 73aba1bfe3054207bba71f56fd9a18b7 (127.28.235.194:40363), f75c23e2671246f78a4fcbf88d48a502 (127.28.235.193:36133)
I20260926 07:22:11.812573 11710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86e2f071c8944ad8e2668e28265be6c" candidate_uuid: "b39868d60a2943e49b38bec1b65ff480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73aba1bfe3054207bba71f56fd9a18b7"
I20260926 07:22:11.812592 11572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86e2f071c8944ad8e2668e28265be6c" candidate_uuid: "b39868d60a2943e49b38bec1b65ff480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75c23e2671246f78a4fcbf88d48a502"
I20260926 07:22:11.812736 11572 raft_consensus.cc:3037] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:11.812750 11710 raft_consensus.cc:3037] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:11.813794 11572 raft_consensus.cc:2454] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b39868d60a2943e49b38bec1b65ff480 in term 1.
I20260926 07:22:11.813989 11710 raft_consensus.cc:2454] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b39868d60a2943e49b38bec1b65ff480 in term 1.
I20260926 07:22:11.814070 11775 leader_election.cc:302] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b39868d60a2943e49b38bec1b65ff480, f75c23e2671246f78a4fcbf88d48a502; no voters:
I20260926 07:22:11.814234 11905 raft_consensus.cc:2781] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:22:11.814462 11905 raft_consensus.cc:686] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 1 LEADER]: Becoming Leader. State: Replica: b39868d60a2943e49b38bec1b65ff480, State: Running, Role: LEADER
I20260926 07:22:11.814805 11905 consensus_queue.cc:237] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:11.818374 11429 catalog_manager.cc:5495] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 reported cstate change: term changed from 0 to 1, leader changed from <none> to b39868d60a2943e49b38bec1b65ff480 (127.28.235.195). New cstate: current_term: 1 leader_uuid: "b39868d60a2943e49b38bec1b65ff480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } health_report { overall_health: UNKNOWN } } }
I20260926 07:22:11.833096 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.829339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.28.235.195:46785)
0926 07:22:11.829512 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:11.829515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:11.829530 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:22:11.829645 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:11.829647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:11.829701 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:11.829840 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:22:11.830086 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.830566 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.831455 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:11.831656 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:11.831731 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:11.832063 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:22:11.832483 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:22:11.832573 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:11.832807 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 07:22:11.832860 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 07:22:11.839830 11572 raft_consensus.cc:1261] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Refusing update from remote peer b39868d60a2943e49b38bec1b65ff480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:11.839869 11710 raft_consensus.cc:1261] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Refusing update from remote peer b39868d60a2943e49b38bec1b65ff480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:11.840577 11909 consensus_queue.cc:1035] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:11.840794 11905 consensus_queue.cc:1035] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:11.846069 11912 mvcc.cc:205] Tried to move back new op lower bound from 7333508431207223296 to 7333508431115980800. Current Snapshot: MvccSnapshot[applied={T|T < 7333508431207223296}]
I20260926 07:22:11.849102 11914 mvcc.cc:205] Tried to move back new op lower bound from 7333508431207223296 to 7333508431115980800. Current Snapshot: MvccSnapshot[applied={T|T < 7333508431207223296}]
I20260926 07:22:11.854929 11429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:22:11.855113 11429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:22:11.857818 11429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Sending DeleteTablet for 3 replicas of tablet e86e2f071c8944ad8e2668e28265be6c
I20260926 07:22:11.858738 11552 tablet_service.cc:1514] Processing DeleteTablet for tablet e86e2f071c8944ad8e2668e28265be6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:11 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:36912
I20260926 07:22:11.858743 11690 tablet_service.cc:1514] Processing DeleteTablet for tablet e86e2f071c8944ad8e2668e28265be6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:11 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:34168
I20260926 07:22:11.858911 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 11468
I20260926 07:22:11.858892 11822 tablet_service.cc:1514] Processing DeleteTablet for tablet e86e2f071c8944ad8e2668e28265be6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:11 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:38794
I20260926 07:22:11.859704 11923 tablet_replica.cc:330] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: stopping tablet replica
I20260926 07:22:11.860082 11923 raft_consensus.cc:2229] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:22:11.860481 11923 raft_consensus.cc:2258] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:11.862233 11923 ts_tablet_manager.cc:1905] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:11.864722 11923 ts_tablet_manager.cc:1918] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:22:11.864841 11923 log.cc:1196] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/e86e2f071c8944ad8e2668e28265be6c
I20260926 07:22:11.865213 11923 ts_tablet_manager.cc:1939] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Deleting consensus metadata
I20260926 07:22:11.865211 11922 tablet_replica.cc:330] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: stopping tablet replica
I20260926 07:22:11.865698 11922 raft_consensus.cc:2229] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:22:11.866190 11922 raft_consensus.cc:2258] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:11.866734 11417 catalog_manager.cc:4841] TS b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785): tablet e86e2f071c8944ad8e2668e28265be6c (table test-table [id=13edce5eaed84b10a354214005c06024]) successfully deleted
W20260926 07:22:11.867398 11414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:36133 (error 108)
I20260926 07:22:11.868003 11922 ts_tablet_manager.cc:1905] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:11.868026 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 11623
I20260926 07:22:11.870783 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.870448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:36133 (local address 127.0.0.1:36920)
0926 07:22:11.870581 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:11.870699 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:36133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:22:11.871011 11414 catalog_manager.cc:4584] TS f75c23e2671246f78a4fcbf88d48a502 (127.28.235.193:36133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e86e2f071c8944ad8e2668e28265be6c: Network error: Client connection negotiation failed: client connection to 127.28.235.193:36133: connect: Connection refused (error 111)
I20260926 07:22:11.876240 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 11759
I20260926 07:22:11.876845 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:11.876473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:40363 (local address 127.0.0.1:34180)
0926 07:22:11.876673 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:11.876738 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.194:40363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 07:22:11.882992 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 11397
I20260926 07:22:11.892094 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:42613
--webserver_interface=127.28.235.254
--webserver_port=38725
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:42613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:12.007732 11925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:12.008049 11925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:12.008127 11925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:12.013188 11925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:22:12.013304 11925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:12.013348 11925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:12.013411 11925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:22:12.013469 11925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:22:12.019986 11925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:42613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:42613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=38725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:12.021267 11925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:12.022400 11925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:22:12.032078 11925 server_base.cc:1034] running on GCE node
W20260926 07:22:12.032083 11933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:12.032022 11930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:12.032021 11931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:12.032833 11925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:12.034103 11925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:12.035321 11925 hybrid_clock.cc:648] HybridClock initialized: now 1790407332035303 us; error 34 us; skew 500 ppm
Sep 26 07:22:12 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407332, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:12.038441 11925 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:22:12.039405 11925 webserver.cc:460] Webserver started at http://127.28.235.254:38725/ using document root <none> and password file <none>
I20260926 07:22:12.039989 11925 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:12.040050 11925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:12.044157 11925 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:22:12.046597 11940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:12.047987 11925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:22:12.048100 11925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "0cf06e20743d46ea86f0f7e477217cf3"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "c93217f68301030105b47b087a031ebb"
server_key_iv: "82122f06b6e21b5eea03ec9074d44897"
server_key_version: "encryptionkey@0"
I20260926 07:22:12.048729 11925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:12.061448 11925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:12.064924 11925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:12.065198 11925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:12.072930 11925 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:42613
I20260926 07:22:12.073305 11992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:42613 every 8 connection(s)
I20260926 07:22:12.074133 11925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:22:12.078737 11993 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:22:12.079432 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 11925
I20260926 07:22:12.079864 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:36133
--local_ip_for_outbound_sockets=127.28.235.193
--tserver_master_addrs=127.28.235.254:42613
--webserver_port=33015
--webserver_interface=127.28.235.193
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:22:12.080608 11993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Bootstrap starting.
I20260926 07:22:12.099043 11993 log.cc:824] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:12.110718 11993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:22:12.111166 11993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3: Bootstrap complete.
I20260926 07:22:12.115607 11993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:12.116062 11993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf06e20743d46ea86f0f7e477217cf3, State: Initialized, Role: FOLLOWER
I20260926 07:22:12.116642 11993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:12.116803 11993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:22:12.116899 11993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:22:12.117056 11993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:22:12.119145 11993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:12.119539 11993 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf06e20743d46ea86f0f7e477217cf3; no voters:
I20260926 07:22:12.119915 11993 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:22:12.120018 11997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:22:12.120220 11997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [term 3 LEADER]: Becoming Leader. State: Replica: 0cf06e20743d46ea86f0f7e477217cf3, State: Running, Role: LEADER
I20260926 07:22:12.120585 11997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } }
I20260926 07:22:12.121265 11993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:22:12.122071 11998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0cf06e20743d46ea86f0f7e477217cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } } }
I20260926 07:22:12.122227 11998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:12.122761 11999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf06e20743d46ea86f0f7e477217cf3. Latest consensus state: current_term: 3 leader_uuid: "0cf06e20743d46ea86f0f7e477217cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06e20743d46ea86f0f7e477217cf3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42613 } } }
I20260926 07:22:12.122856 11999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf06e20743d46ea86f0f7e477217cf3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:12.123726 12004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:22:12.129073 12004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:22:12.129830 12004 catalog_manager.cc:1259] Loaded cluster ID: b13c0a8bf98f445583a863a4141052b9
I20260926 07:22:12.129899 12004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 07:22:12.130937 12004 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f84e3ee59ad google::LogMessage::Fail() at ??:0
@ 0x7f84e3ee888a google::LogMessage::SendToLog() at ??:0
@ 0x7f84e3ee54de google::LogMessage::Flush() at ??:0
@ 0x7f84e3ee6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f84ee824fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f84ee825d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f84ee822579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f84ee866640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f84e61373d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f84e6137c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f84e61393f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f84e6128512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f84e776c6db start_thread at ??:0
@ 0x7f84e29e361f clone at ??:0
W20260926 07:22:12.215134 11995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:12.215524 11995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:12.215619 11995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:12.219919 11995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:12.220021 11995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:12.220129 11995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:22:12.226082 11995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:36133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=33015
--tserver_master_addrs=127.28.235.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:12.227313 11995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:12.228407 11995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:12.239131 12036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:12.239195 12038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:12.239146 12035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:12.239676 11995 server_base.cc:1034] running on GCE node
I20260926 07:22:12.240134 11995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:12.240880 11995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:12.242108 11995 hybrid_clock.cc:648] HybridClock initialized: now 1790407332242083 us; error 41 us; skew 500 ppm
Sep 26 07:22:12 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407332, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:12.245386 11995 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:22:12.246572 11995 webserver.cc:460] Webserver started at http://127.28.235.193:33015/ using document root <none> and password file <none>
I20260926 07:22:12.247241 11995 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:12.247310 11995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:12.251999 11995 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:22:12.254756 12045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:12.256579 11995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:22:12.256775 11995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "f75c23e2671246f78a4fcbf88d48a502"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "9f331d7d686a4facd8f8b86c68a6df09"
server_key_iv: "7b832d5fb05e5a4a30312deacfb9a927"
server_key_version: "encryptionkey@0"
I20260926 07:22:12.257555 11995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:12.274255 11995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:12.278056 11995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:12.278291 11995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:12.278951 11995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:12.280118 12052 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:22:12.285983 11995 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:22:12.286095 11995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 07:22:12.286201 11995 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:22:12.288136 11995 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:22:12.288297 11995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 07:22:12.288550 12052 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Bootstrap starting.
I20260926 07:22:12.300019 11995 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:36133
I20260926 07:22:12.300199 12159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:36133 every 8 connection(s)
I20260926 07:22:12.301156 11995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:22:12.307139 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 11995
I20260926 07:22:12.307586 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:40363
--local_ip_for_outbound_sockets=127.28.235.194
--tserver_master_addrs=127.28.235.254:42613
--webserver_port=36841
--webserver_interface=127.28.235.194
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:22:12.311856 12052 log.cc:824] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:12.321866 12052 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:22:12.322355 12052 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Bootstrap complete.
I20260926 07:22:12.323961 12052 ts_tablet_manager.cc:1397] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Time spent bootstrapping tablet: real 0.036s user 0.021s sys 0.011s
I20260926 07:22:12.327641 12052 raft_consensus.cc:348] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:12.328305 12052 raft_consensus.cc:729] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75c23e2671246f78a4fcbf88d48a502, State: Initialized, Role: FOLLOWER
I20260926 07:22:12.328999 12052 consensus_queue.cc:260] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:12.330212 12052 ts_tablet_manager.cc:1428] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
W20260926 07:22:12.409590 12162 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:22:12.303319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:59795)
0926 07:22:12.303811 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.303867 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:12.304770 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:12.408524 (+103754us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":146,"threads_started":1}
W20260926 07:22:12.410298 12160 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:42613 (0 consecutive failures): Network error: Failed to ping master at 127.28.235.254:42613: Client connection negotiation failed: client connection to 127.28.235.254:42613: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:22:12.411199 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:12.410889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:48855)
0926 07:22:12.411045 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.411127 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 07:22:12.411964 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:12.411723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:56889)
0926 07:22:12.411851 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.411904 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:22:12.412271 12160 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:22:12.431203 12163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:12.431493 12163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:12.431568 12163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:12.435863 12163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:12.435923 12163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:12.436033 12163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:22:12.441463 12163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:40363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=36841
--tserver_master_addrs=127.28.235.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:12.442540 12163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:12.443574 12163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:12.453405 12171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:12.453405 12170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:12.453408 12173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:12.453639 12163 server_base.cc:1034] running on GCE node
I20260926 07:22:12.454221 12163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:12.454918 12163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:12.456115 12163 hybrid_clock.cc:648] HybridClock initialized: now 1790407332456079 us; error 47 us; skew 500 ppm
Sep 26 07:22:12 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407332, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:12.459050 12163 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:22:12.459954 12163 webserver.cc:460] Webserver started at http://127.28.235.194:36841/ using document root <none> and password file <none>
I20260926 07:22:12.460484 12163 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:12.460544 12163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:12.464608 12163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:22:12.466967 12180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:12.468199 12163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:22:12.468299 12163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "73aba1bfe3054207bba71f56fd9a18b7"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "6c32e80aa3c8f30ef604bc6bbbeb8371"
server_key_iv: "97451c5f254690d34561826cb82a4bfd"
server_key_version: "encryptionkey@0"
I20260926 07:22:12.468871 12163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:12.495167 12163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:12.498672 12163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:12.498842 12163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:12.499396 12163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:12.500443 12187 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:22:12.504678 12163 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:22:12.504742 12163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 07:22:12.504871 12163 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:22:12.506533 12163 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:22:12.506575 12163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 07:22:12.506915 12187 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Bootstrap starting.
I20260926 07:22:12.512740 12187 log.cc:824] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:12.517336 12163 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:40363
I20260926 07:22:12.517371 12295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:40363 every 8 connection(s)
I20260926 07:22:12.518167 12163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:22:12.521327 12187 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:22:12.521483 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:12.520113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:40703)
0926 07:22:12.520609 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.520722 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":159,"threads_started":1}
I20260926 07:22:12.521811 12187 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Bootstrap complete.
W20260926 07:22:12.521919 12296 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:42613 (0 consecutive failures): Network error: Failed to ping master at 127.28.235.254:42613: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
I20260926 07:22:12.522173 12187 ts_tablet_manager.cc:1397] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260926 07:22:12.522523 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:12.522312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:51067)
0926 07:22:12.522430 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.522474 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 07:22:12.523188 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:12.523005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:60055)
0926 07:22:12.523103 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.523143 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:22:12.523499 12296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:22:12.525264 12187 raft_consensus.cc:348] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:12.525702 12187 raft_consensus.cc:729] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73aba1bfe3054207bba71f56fd9a18b7, State: Initialized, Role: FOLLOWER
I20260926 07:22:12.526145 12187 consensus_queue.cc:260] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:12.527101 12187 ts_tablet_manager.cc:1428] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 07:22:12.527117 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:12.526915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:46093)
0926 07:22:12.527019 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.527071 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 07:22:12.527283 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 12163
I20260926 07:22:12.527666 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:46785
--local_ip_for_outbound_sockets=127.28.235.195
--tserver_master_addrs=127.28.235.254:42613
--webserver_port=40631
--webserver_interface=127.28.235.195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:22:12.527797 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:12.527586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:42855)
0926 07:22:12.527687 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.527737 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 07:22:12.643903 12300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:12.644194 12300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:12.644238 12300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:12.648362 12300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:12.648427 12300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:12.648522 12300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:22:12.653995 12300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:46785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=40631
--tserver_master_addrs=127.28.235.254:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:12.655148 12300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:12.656225 12300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:12.666422 12308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:12.666422 12306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:12.666442 12305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:12.667071 12300 server_base.cc:1034] running on GCE node
I20260926 07:22:12.667616 12300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:12.668385 12300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:12.669724 12300 hybrid_clock.cc:648] HybridClock initialized: now 1790407332669635 us; error 53 us; skew 500 ppm
Sep 26 07:22:12 dist-test-slave-7zlw krb5kdc[10848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407332, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:12.672889 12300 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:22:12.673980 12300 webserver.cc:460] Webserver started at http://127.28.235.195:40631/ using document root <none> and password file <none>
I20260926 07:22:12.674546 12300 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:12.674608 12300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:12.678746 12300 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:22:12.680962 12315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:12.682374 12300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:12.682482 12300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "b39868d60a2943e49b38bec1b65ff480"
format_stamp: "Formatted at 2026-09-26 07:22:09 on dist-test-slave-7zlw"
server_key: "fc5882a3b22711067cd93ff4de84ba24"
server_key_iv: "9a2e00b7c468b62248ab8bc592b9df21"
server_key_version: "encryptionkey@0"
I20260926 07:22:12.683073 12300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:12.695569 12300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:12.699036 12300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:12.699222 12300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:12.699838 12300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:12.700724 12300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:12.700781 12300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:12.700827 12300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:12.700845 12300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:12.711144 12300 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:46785
I20260926 07:22:12.711163 12428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:46785 every 8 connection(s)
I20260926 07:22:12.712322 12300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:22:12.714504 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 12300
I20260926 07:22:12.716835 12431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:12.714897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:35757)
0926 07:22:12.715432 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.715575 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":155,"threads_started":1}
W20260926 07:22:12.717588 12429 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:42613 (0 consecutive failures): Network error: Failed to ping master at 127.28.235.254:42613: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
I20260926 07:22:12.718415 12431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:12.718165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:45421)
0926 07:22:12.718295 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.718352 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:22:12.719076 12431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:12.718856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:45155)
0926 07:22:12.718970 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:12.719028 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:22:12.719259 12429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:22:13.413462 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:13.412975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:45629)
0926 07:22:13.413286 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:13.413350 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":128,"threads_started":1}
I20260926 07:22:13.528896 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:13.528493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:36271)
0926 07:22:13.528741 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:13.528800 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":66,"threads_started":1}
I20260926 07:22:13.640789 12435 raft_consensus.cc:482] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:22:13.641088 12435 raft_consensus.cc:504] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:13.642272 12435 leader_election.cc:288] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73aba1bfe3054207bba71f56fd9a18b7 (127.28.235.194:40363), b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785)
Sep 26 07:22:13 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407332, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.195@KRBTEST.COM
Sep 26 07:22:13 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407332, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.194@KRBTEST.COM
I20260926 07:22:13.656664 12438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:13.642759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:38527 (local address 127.28.235.195:46785)
0926 07:22:13.643118 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:13.643137 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:13.643188 (+ 51us) server_negotiation.cc:407] Connection header received
0926 07:22:13.644041 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:13.644049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:13.644368 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:13.644627 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:13.645714 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:13.646712 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:13.647985 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:13.648498 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:13.648668 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:13.651766 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:13.651778 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:13.651800 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:13.651843 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:13.653878 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:13.654551 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:13.654557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:13.654563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:13.654721 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:13.655136 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:13.655139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:13.655141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:13.655554 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:13.655699 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:13.656224 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 07:22:13.656499 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":90,"threads_started":1}
I20260926 07:22:13.656646 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:13.642595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:46785 (local address 127.28.235.193:38527)
0926 07:22:13.642940 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:13.642956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:13.643060 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:13.644573 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:13.644576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:13.645034 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:13.645534 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:13.645549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:13.646863 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:13.646867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:13.647855 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:13.647865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:13.648619 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:13.648622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:13.648859 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:13.649681 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:13.649698 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:13.651490 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:13.654006 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:13.654012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:13.654026 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:13.654442 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:13.654839 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:13.654841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:13.654843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:13.655021 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:13.655673 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:13.655679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:13.655918 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 07:22:13.656101 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 07:22:13.656446 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":170,"spinlock_wait_cycles":29696,"thread_start_us":59,"threads_started":1}
I20260926 07:22:13.656792 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:13.642685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:53903 (local address 127.28.235.194:40363)
0926 07:22:13.643169 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:13.643180 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:13.643212 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:22:13.643531 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:13.643538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:13.644002 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:13.644387 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:13.645710 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:13.646732 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:13.647998 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:13.648500 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:13.648683 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:13.652034 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:13.652039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:13.652050 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:13.652077 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:13.654002 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:13.654576 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:13.654581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:13.654586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:13.654721 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:13.655154 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:13.655157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:13.655159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:13.655579 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:13.655737 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:13.656238 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 07:22:13.656554 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":72,"threads_started":1}
I20260926 07:22:13.657158 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:13.642563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:40363 (local address 127.28.235.193:53903)
0926 07:22:13.642766 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:13.642783 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:13.642899 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:13.644428 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:13.644434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:13.645001 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:13.645534 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:13.645549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:13.646879 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:13.646883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:13.647858 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:13.647868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:13.648635 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:13.648639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:13.648867 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:13.649744 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:13.649758 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:13.651879 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:13.654116 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:13.654118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:13.654120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:13.654474 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:13.654869 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:13.654875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:13.654879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:13.655020 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:13.655702 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:13.655706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:13.655924 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 07:22:13.656102 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 07:22:13.656554 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":239,"spinlock_wait_cycles":68224}
I20260926 07:22:13.658339 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86e2f071c8944ad8e2668e28265be6c" candidate_uuid: "f75c23e2671246f78a4fcbf88d48a502" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "73aba1bfe3054207bba71f56fd9a18b7" is_pre_election: true
I20260926 07:22:13.658668 12249 raft_consensus.cc:2454] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f75c23e2671246f78a4fcbf88d48a502 in term 1.
I20260926 07:22:13.659592 12048 leader_election.cc:302] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73aba1bfe3054207bba71f56fd9a18b7, f75c23e2671246f78a4fcbf88d48a502; no voters:
W20260926 07:22:13.659777 12049 leader_election.cc:341] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785): Not found: Tablet not found: e86e2f071c8944ad8e2668e28265be6c
I20260926 07:22:13.659916 12435 raft_consensus.cc:2781] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 07:22:13.660037 12435 raft_consensus.cc:482] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:22:13.660100 12435 raft_consensus.cc:3037] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:22:13.662576 12435 raft_consensus.cc:504] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:13.662994 12435 leader_election.cc:288] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [CANDIDATE]: Term 2 election: Requested vote from peers 73aba1bfe3054207bba71f56fd9a18b7 (127.28.235.194:40363), b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785)
I20260926 07:22:13.663496 12249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e86e2f071c8944ad8e2668e28265be6c" candidate_uuid: "f75c23e2671246f78a4fcbf88d48a502" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "73aba1bfe3054207bba71f56fd9a18b7"
I20260926 07:22:13.663661 12249 raft_consensus.cc:3037] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 1 FOLLOWER]: Advancing to term 2
W20260926 07:22:13.663741 12049 leader_election.cc:341] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785): Not found: Tablet not found: e86e2f071c8944ad8e2668e28265be6c
I20260926 07:22:13.666316 12249 raft_consensus.cc:2454] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f75c23e2671246f78a4fcbf88d48a502 in term 2.
I20260926 07:22:13.666678 12048 leader_election.cc:302] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73aba1bfe3054207bba71f56fd9a18b7, f75c23e2671246f78a4fcbf88d48a502; no voters: b39868d60a2943e49b38bec1b65ff480
I20260926 07:22:13.667016 12435 raft_consensus.cc:2781] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:22:13.667183 12435 raft_consensus.cc:686] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [term 2 LEADER]: Becoming Leader. State: Replica: f75c23e2671246f78a4fcbf88d48a502, State: Running, Role: LEADER
I20260926 07:22:13.667587 12435 consensus_queue.cc:237] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:13.667948 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:13.667657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:43923)
0926 07:22:13.667808 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:13.667866 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 07:22:13.720414 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:13.719919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:34557)
0926 07:22:13.720238 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:13.720303 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260926 07:22:14.066121 12249 raft_consensus.cc:1261] T e86e2f071c8944ad8e2668e28265be6c P 73aba1bfe3054207bba71f56fd9a18b7 [term 2 FOLLOWER]: Refusing update from remote peer f75c23e2671246f78a4fcbf88d48a502: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 07:22:14.066886 12435 consensus_queue.cc:1035] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 07:22:14.069064 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:14.068544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:41657)
0926 07:22:14.068833 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:14.068910 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":66,"threads_started":1}
I20260926 07:22:14.071465 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:14.071152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:47691)
0926 07:22:14.071304 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:14.071370 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260926 07:22:14.071686 12049 consensus_peers.cc:487] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502 -> Peer b39868d60a2943e49b38bec1b65ff480 (127.28.235.195:46785): Couldn't send request to peer b39868d60a2943e49b38bec1b65ff480. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e86e2f071c8944ad8e2668e28265be6c. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:22:14.072140 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:14.071890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:33485)
0926 07:22:14.072003 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:14.072083 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 07:22:14.661752 12449 ts_tablet_manager.cc:927] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Initiating tablet copy from peer f75c23e2671246f78a4fcbf88d48a502 (127.28.235.193:36133)
I20260926 07:22:14.662943 12449 tablet_copy_client.cc:316] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: tablet copy: Beginning tablet copy session from remote peer at address 127.28.235.193:36133
Sep 26 07:22:14 dist-test-slave-7zlw krb5kdc[10848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407332, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:22:14.681734 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:14.663873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:35631 (local address 127.28.235.193:36133)
0926 07:22:14.664210 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:14.664217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:14.664523 (+ 306us) server_negotiation.cc:407] Connection header received
0926 07:22:14.664676 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:14.664685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:14.664918 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:14.665231 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:14.666159 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:14.666985 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:14.668068 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:14.668523 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:14.668622 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:14.676837 (+ 8215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:14.676854 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:14.676871 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:14.676911 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:14.679173 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:14.679962 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:14.679971 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:14.679973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:14.680077 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:14.680409 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:14.680414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:14.680416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:14.680725 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:14.680889 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:14.681378 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 07:22:14.681542 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260926 07:22:14.681736 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:14.663675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:36133 (local address 127.28.235.195:35631)
0926 07:22:14.664296 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:14.664320 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:14.664547 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:14.665254 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:14.665281 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:14.665837 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:14.665998 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:14.666005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:14.667132 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:14.667134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:14.667925 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:14.667935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:14.668660 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:14.668663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:14.668785 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:14.674656 (+ 5871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:14.674676 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:14.676487 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:14.679403 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:14.679421 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:14.679428 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:14.679786 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:14.680196 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:14.680202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:14.680204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:14.680303 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:14.680860 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:14.680869 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:14.681091 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 07:22:14.681272 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 07:22:14.681508 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":147,"threads_started":1}
I20260926 07:22:14.683364 12134 tablet_copy_service.cc:140] P f75c23e2671246f78a4fcbf88d48a502: Received BeginTabletCopySession request for tablet e86e2f071c8944ad8e2668e28265be6c from peer b39868d60a2943e49b38bec1b65ff480 ({username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:35631)
I20260926 07:22:14.683537 12134 tablet_copy_service.cc:161] P f75c23e2671246f78a4fcbf88d48a502: Beginning new tablet copy session on tablet e86e2f071c8944ad8e2668e28265be6c from peer b39868d60a2943e49b38bec1b65ff480 at {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:35631: session id = b39868d60a2943e49b38bec1b65ff480-e86e2f071c8944ad8e2668e28265be6c
I20260926 07:22:14.684777 12134 tablet_copy_source_session.cc:205] T e86e2f071c8944ad8e2668e28265be6c P f75c23e2671246f78a4fcbf88d48a502: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:22:14.689811 12449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e86e2f071c8944ad8e2668e28265be6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:14.698271 12449 tablet_copy_client.cc:791] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: tablet copy: Starting download of 0 data blocks...
I20260926 07:22:14.698550 12449 tablet_copy_client.cc:655] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: tablet copy: Starting download of 1 WAL segments...
I20260926 07:22:14.701100 12449 tablet_copy_client.cc:531] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:22:14.703745 12449 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Bootstrap starting.
I20260926 07:22:14.712077 12449 log.cc:824] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:14.721858 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:14.721460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:60973)
0926 07:22:14.721643 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:14.721722 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 07:22:14.730684 12449 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:22:14.731189 12449 tablet_bootstrap.cc:492] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Bootstrap complete.
I20260926 07:22:14.731703 12449 ts_tablet_manager.cc:1397] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.005s
I20260926 07:22:14.736189 12449 raft_consensus.cc:348] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:14.736618 12449 raft_consensus.cc:729] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39868d60a2943e49b38bec1b65ff480, State: Initialized, Role: FOLLOWER
I20260926 07:22:14.737339 12449 consensus_queue.cc:260] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73aba1bfe3054207bba71f56fd9a18b7" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 40363 } } peers { permanent_uuid: "b39868d60a2943e49b38bec1b65ff480" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46785 } } peers { permanent_uuid: "f75c23e2671246f78a4fcbf88d48a502" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36133 } }
I20260926 07:22:14.738860 12449 ts_tablet_manager.cc:1428] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260926 07:22:14.739116 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:14.738771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:58367)
0926 07:22:14.738916 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:14.738994 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 07:22:14.739859 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:14.739616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:58325)
0926 07:22:14.739742 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:14.739795 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 07:22:14.739909 12134 tablet_copy_service.cc:342] P f75c23e2671246f78a4fcbf88d48a502: Request end of tablet copy session b39868d60a2943e49b38bec1b65ff480-e86e2f071c8944ad8e2668e28265be6c received from {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:35631
I20260926 07:22:14.740046 12134 tablet_copy_service.cc:433] P f75c23e2671246f78a4fcbf88d48a502: ending tablet copy session b39868d60a2943e49b38bec1b65ff480-e86e2f071c8944ad8e2668e28265be6c on tablet e86e2f071c8944ad8e2668e28265be6c with peer b39868d60a2943e49b38bec1b65ff480
I20260926 07:22:15.073066 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:15.072376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:50597)
0926 07:22:15.072734 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:15.072875 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":72,"threads_started":1}
I20260926 07:22:15.073377 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:15.072857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:52451)
0926 07:22:15.073146 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:15.073244 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260926 07:22:15.090515 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:15.090086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:53073)
0926 07:22:15.090281 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:15.090380 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 07:22:15.090875 12383 raft_consensus.cc:1203] T e86e2f071c8944ad8e2668e28265be6c P b39868d60a2943e49b38bec1b65ff480 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 07:22:15.092006 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:15.091696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:35317)
0926 07:22:15.091833 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:15.091920 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 07:22:16.074805 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:16.074307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:48071)
0926 07:22:16.074633 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:16.074696 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":76,"threads_started":1}
I20260926 07:22:16.091853 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:16.091384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:43061)
0926 07:22:16.091677 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:16.091749 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":88,"threads_started":1}
I20260926 07:22:16.093262 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:16.092792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:36927)
0926 07:22:16.093120 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:16.093169 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
I20260926 07:22:17.076077 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:17.075577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:42653)
0926 07:22:17.075887 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:17.075971 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260926 07:22:17.093293 12462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:17.092755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:55899)
0926 07:22:17.093098 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:17.093186 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260926 07:22:17.094410 12463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:17.094040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:53947)
0926 07:22:17.094278 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:17.094323 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":57,"threads_started":1}
I20260926 07:22:18.077682 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:18.077259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:56713)
0926 07:22:18.077523 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:18.077586 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260926 07:22:18.094614 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:18.094221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:35991)
0926 07:22:18.094466 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:18.094521 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
I20260926 07:22:18.095299 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:18.095072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:50661)
0926 07:22:18.095206 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:18.095247 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:22:18.095686 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:18.095310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:41963)
0926 07:22:18.095564 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:18.095611 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260926 07:22:19.079169 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:19.078588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:44477)
0926 07:22:19.078989 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:19.079052 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":219,"threads_started":1}
I20260926 07:22:19.096689 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:19.096093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:54229)
0926 07:22:19.096419 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:19.096497 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
I20260926 07:22:19.096802 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:19.096400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:47257)
0926 07:22:19.096625 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:19.096682 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
I20260926 07:22:20.080495 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:20.080020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:60079)
0926 07:22:20.080338 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:20.080398 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":96,"threads_started":1}
I20260926 07:22:20.098109 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:20.097622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:37155)
0926 07:22:20.097929 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:20.098007 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260926 07:22:20.098107 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:20.097623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:34043)
0926 07:22:20.097964 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:20.098019 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":147,"threads_started":1}
I20260926 07:22:21.082223 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:21.081697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:47529)
0926 07:22:21.082017 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:21.082085 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260926 07:22:21.099434 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:21.098988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:35115)
0926 07:22:21.099292 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:21.099344 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":121,"threads_started":1}
I20260926 07:22:21.099472 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:21.098985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:38225)
0926 07:22:21.099324 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:21.099389 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260926 07:22:22.083711 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:22.083239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:49237)
0926 07:22:22.083536 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:22.083599 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":115,"threads_started":1}
I20260926 07:22:22.100829 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:22.100320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:59995)
0926 07:22:22.100646 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:22.100713 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":90,"threads_started":1}
I20260926 07:22:22.100828 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:22.100320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:42953)
0926 07:22:22.100646 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:22.100713 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":92,"threads_started":1}
I20260926 07:22:23.085237 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:23.084663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:59341)
0926 07:22:23.085007 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:23.085141 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260926 07:22:23.102399 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:23.101948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:50123)
0926 07:22:23.102225 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:23.102277 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
I20260926 07:22:23.102401 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:23.101877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:38077)
0926 07:22:23.102172 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:23.102247 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260926 07:22:24.086460 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:24.086012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:37821)
0926 07:22:24.086298 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:24.086363 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260926 07:22:24.103632 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:24.103199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:59915)
0926 07:22:24.103474 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:24.103537 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
I20260926 07:22:24.103647 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:24.103255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:35215)
0926 07:22:24.103504 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:24.103557 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260926 07:22:25.088197 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:25.087600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:59869)
0926 07:22:25.087997 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:25.088090 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":99,"threads_started":1}
I20260926 07:22:25.105252 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:25.104673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:60687)
0926 07:22:25.104987 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:25.105114 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":124,"threads_started":1}
I20260926 07:22:25.105417 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:25.104871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:48493)
0926 07:22:25.105217 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:25.105278 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260926 07:22:26.089515 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:26.089083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:41853)
0926 07:22:26.089352 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:26.089415 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260926 07:22:26.106617 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:26.106142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:51981)
0926 07:22:26.106443 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:26.106516 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":75,"threads_started":1}
I20260926 07:22:26.106695 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:26.106209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:42175)
0926 07:22:26.106560 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:26.106613 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":99,"threads_started":1}
I20260926 07:22:27.090832 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:27.090342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.194:45467)
0926 07:22:27.090600 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:27.090727 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260926 07:22:27.108088 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:27.107580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.193:46965)
0926 07:22:27.107882 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:27.107962 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260926 07:22:27.108170 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:27.107630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:42613 (local address 127.28.235.195:39817)
0926 07:22:27.107998 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:27.108071 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:42613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":92,"threads_started":1}
I20260926 07:22:27.717298 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 11995
I20260926 07:22:27.723291 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 12163
I20260926 07:22:27.729683 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 12300
2026-09-26T07:22:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19829 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:22:27 dist-test-slave-7zlw krb5kdc[12501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:22:27 dist-test-slave-7zlw krb5kdc[12501](info): set up 2 sockets
Sep 26 07:22:27 dist-test-slave-7zlw krb5kdc[12501](info): commencing operation
krb5kdc: starting...
W20260926 07:22:28.793454 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:22:28 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:22:28.924865 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43069
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:29.044817 12514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:29.045243 12514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:29.045359 12514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:29.049611 12514 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:22:29.049724 12514 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:22:29.049750 12514 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:22:29.049772 12514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:22:29.049817 12514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:29.049875 12514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:29.049908 12514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:22:29.049947 12514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:22:29.055930 12514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:43069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:29.057157 12514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:29.058233 12514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:29.067627 12519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:29.067689 12514 server_base.cc:1034] running on GCE node
W20260926 07:22:29.067627 12522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:29.067663 12520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:29.068686 12514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:29.069974 12514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:29.071167 12514 hybrid_clock.cc:648] HybridClock initialized: now 1790407349071152 us; error 30 us; skew 500 ppm
Sep 26 07:22:29 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407349, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:29.074397 12514 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:22:29.075385 12514 webserver.cc:460] Webserver started at http://127.28.235.254:35625/ using document root <none> and password file <none>
I20260926 07:22:29.076074 12514 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:29.076140 12514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:29.076364 12514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:29.078632 12514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "2c536ffc3b9744aeafa34fd4c16c96f9"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "adb15f9f33f67541e4bfd68288823cb3"
server_key_iv: "7b6fe4491861fba90000500b84e5f062"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.079245 12514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "2c536ffc3b9744aeafa34fd4c16c96f9"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "adb15f9f33f67541e4bfd68288823cb3"
server_key_iv: "7b6fe4491861fba90000500b84e5f062"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.083395 12514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:22:29.085973 12529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:29.087415 12514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:22:29.087572 12514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "2c536ffc3b9744aeafa34fd4c16c96f9"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "adb15f9f33f67541e4bfd68288823cb3"
server_key_iv: "7b6fe4491861fba90000500b84e5f062"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.087704 12514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:29.110224 12514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:29.113956 12514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:29.114187 12514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:29.121609 12514 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:43069
I20260926 07:22:29.121937 12581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:43069 every 8 connection(s)
I20260926 07:22:29.122676 12514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:22:29.125769 12582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:29.131590 12582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Bootstrap starting.
I20260926 07:22:29.132114 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 12514
I20260926 07:22:29.132265 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:22:29.132548 29615 external_mini_cluster.cc:1428] Setting key 879b75b519dc5f6bce95fca8a2a81699
I20260926 07:22:29.134802 12582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:29.135658 12582 log.cc:824] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:29.138053 12582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: No bootstrap required, opened a new log
Sep 26 07:22:29 dist-test-slave-7zlw krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:29.142844 12582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:22:29.143112 12582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:29.143196 12582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c536ffc3b9744aeafa34fd4c16c96f9, State: Initialized, Role: FOLLOWER
I20260926 07:22:29.143733 12582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:22:29.143896 12582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:22:29.143990 12582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:22:29.144104 12582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:29.145148 12582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:22:29.145525 12582 leader_election.cc:302] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c536ffc3b9744aeafa34fd4c16c96f9; no voters:
I20260926 07:22:29.145897 12582 leader_election.cc:288] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:22:29.146005 12587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:22:29.146216 12587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 1 LEADER]: Becoming Leader. State: Replica: 2c536ffc3b9744aeafa34fd4c16c96f9, State: Running, Role: LEADER
I20260926 07:22:29.146592 12587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:22:29.147374 12582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:22:29.148352 12589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c536ffc3b9744aeafa34fd4c16c96f9. Latest consensus state: current_term: 1 leader_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } } }
I20260926 07:22:29.148483 12589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:29.148785 12588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } } }
I20260926 07:22:29.148862 12588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:29.148895 12596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:22:29.148922 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.134083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.28.235.254:43069)
0926 07:22:29.134539 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:29.134555 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:29.134592 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:22:29.135321 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:29.135341 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:29.135675 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:29.136008 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:29.136914 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.137974 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.139148 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.139645 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.139770 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.142947 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:29.142952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:29.142967 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:29.142996 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:29.144838 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:29.145324 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:29.145331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:29.145335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:29.145447 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:29.145746 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:29.145751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:29.145754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:29.146333 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:29.147341 (+ 1008us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:29.147512 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 07:22:29.147840 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":151,"threads_started":1}
I20260926 07:22:29.152104 12596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:22:29.158723 12596 catalog_manager.cc:1347] Generated new cluster ID: ce42ce341f98431bbb26f34b4cac36d6
I20260926 07:22:29.158819 12596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:22:29.166929 12596 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:22:29.168233 12596 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:22:29.174023 12596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Generated new TSK 0
I20260926 07:22:29.174889 12596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:22:29.209177 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43069
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:29.325315 12608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:29.325583 12608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:29.325627 12608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:29.329728 12608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:29.329793 12608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:29.329902 12608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:22:29.335415 12608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:29.336519 12608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:29.337596 12608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:29.347334 12613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:29.347268 12616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:29.347586 12608 server_base.cc:1034] running on GCE node
W20260926 07:22:29.347450 12614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:29.348131 12608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:29.348875 12608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:29.350086 12608 hybrid_clock.cc:648] HybridClock initialized: now 1790407349350063 us; error 41 us; skew 500 ppm
Sep 26 07:22:29 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407349, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:29.353101 12608 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:22:29.354058 12608 webserver.cc:460] Webserver started at http://127.28.235.193:42631/ using document root <none> and password file <none>
I20260926 07:22:29.354753 12608 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:29.354858 12608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:29.355165 12608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:29.357364 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "f1ca4960845f43f8b0c66d5ca6074c84"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "fe500d24e310c732ed99a38ce8f49a91"
server_key_iv: "fc876c52eaae92c3c731c57d5c4ef456"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.357991 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "f1ca4960845f43f8b0c66d5ca6074c84"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "fe500d24e310c732ed99a38ce8f49a91"
server_key_iv: "fc876c52eaae92c3c731c57d5c4ef456"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.361840 12608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:22:29.364254 12623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:29.365556 12608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:22:29.365674 12608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "f1ca4960845f43f8b0c66d5ca6074c84"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "fe500d24e310c732ed99a38ce8f49a91"
server_key_iv: "fc876c52eaae92c3c731c57d5c4ef456"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.365787 12608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:29.399528 12608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:29.402457 12608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:29.402611 12608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:29.403136 12608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:29.403913 12608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:29.403960 12608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:29.403999 12608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:29.404014 12608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:29.413261 12608 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:38343
I20260926 07:22:29.413278 12736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:38343 every 8 connection(s)
I20260926 07:22:29.414162 12608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:22:29.416327 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 12608
I20260926 07:22:29.416458 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:22:29.416788 29615 external_mini_cluster.cc:1428] Setting key d47a270ec93aed18c7b389a6c2deb0bb
Sep 26 07:22:29 dist-test-slave-7zlw krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407349, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:22:29.428676 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.416000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:49775 (local address 127.28.235.254:43069)
0926 07:22:29.416130 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:29.416133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:29.417571 (+ 1438us) server_negotiation.cc:407] Connection header received
0926 07:22:29.418650 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:29.418654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:29.418709 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:29.418787 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:29.420041 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.420587 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.421840 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.422087 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.422201 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.425005 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:29.425008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:29.425055 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:29.425092 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:29.426640 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:29.427179 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:29.427182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:29.427183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:29.427271 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:29.427580 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:29.427582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:29.427583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:29.427765 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:29.427853 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:29.428419 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 07:22:29.428529 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:22:29.429589 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.416451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:49775)
0926 07:22:29.417403 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:29.417467 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:29.418442 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:29.418970 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:29.418977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:29.419342 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:29.419887 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.419901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.420719 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.420723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.421703 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.421713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.422248 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.422252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.422426 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.423109 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:29.423126 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:29.424788 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:29.426763 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:29.426769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:29.426785 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:29.427075 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:29.427365 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:29.427367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:29.427368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:29.427502 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:29.427888 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:29.427893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:29.428195 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 07:22:29.428358 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 07:22:29.428633 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":159,"threads_started":1}
I20260926 07:22:29.430953 12737 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43069
I20260926 07:22:29.431213 12737 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:29.431787 12737 heartbeater.cc:511] Master 127.28.235.254:43069 requested a full tablet report, sending...
I20260926 07:22:29.433670 12546 ts_manager.cc:194] Registered new tserver with Master: f1ca4960845f43f8b0c66d5ca6074c84 (127.28.235.193:38343)
I20260926 07:22:29.435427 12546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:49775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:22:29.448997 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43069
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:29.561605 12742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:29.561889 12742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:29.561934 12742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:29.565929 12742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:29.565991 12742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:29.566051 12742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:22:29.571604 12742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:29.572708 12742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:29.573742 12742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:29.583726 12748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:29.583832 12747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:29.583850 12742 server_base.cc:1034] running on GCE node
W20260926 07:22:29.584046 12750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:29.584483 12742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:29.585297 12742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:29.586480 12742 hybrid_clock.cc:648] HybridClock initialized: now 1790407349586437 us; error 50 us; skew 500 ppm
Sep 26 07:22:29 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407349, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:29.589623 12742 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:22:29.590574 12742 webserver.cc:460] Webserver started at http://127.28.235.194:45703/ using document root <none> and password file <none>
I20260926 07:22:29.591203 12742 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:29.591264 12742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:29.591439 12742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:29.593677 12742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "a61de86d26384c678c85445894340a2d"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "7688e30ef3a20d7a355ee8b3d7ce08fa"
server_key_iv: "5617fe176e035f1b7f64a8be58f4e9d5"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.594262 12742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "a61de86d26384c678c85445894340a2d"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "7688e30ef3a20d7a355ee8b3d7ce08fa"
server_key_iv: "5617fe176e035f1b7f64a8be58f4e9d5"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.598093 12742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:22:29.600551 12757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:29.601912 12742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:22:29.602052 12742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "a61de86d26384c678c85445894340a2d"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "7688e30ef3a20d7a355ee8b3d7ce08fa"
server_key_iv: "5617fe176e035f1b7f64a8be58f4e9d5"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.602173 12742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:29.617631 12742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:29.620939 12742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:29.621179 12742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:29.621793 12742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:29.622640 12742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:29.622715 12742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:29.622785 12742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:29.622824 12742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:29.632023 12742 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:44013
I20260926 07:22:29.632057 12870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:44013 every 8 connection(s)
I20260926 07:22:29.632936 12742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:22:29.635778 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 12742
I20260926 07:22:29.635901 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:22:29.636217 29615 external_mini_cluster.cc:1428] Setting key 5ca2c924d98827501f74c299fde422d0
Sep 26 07:22:29 dist-test-slave-7zlw krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407349, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:22:29.647645 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.634881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:57791 (local address 127.28.235.254:43069)
0926 07:22:29.635028 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:29.635032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:29.636120 (+ 1088us) server_negotiation.cc:407] Connection header received
0926 07:22:29.637237 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:29.637240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:29.637283 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:29.637357 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:29.638622 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.639142 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.640282 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.640669 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.640814 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.643433 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:29.643438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:29.643444 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:29.643479 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:29.645315 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:29.645875 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:29.645880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:29.645884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:29.645981 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:29.646397 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:29.646399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:29.646401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:29.646589 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:29.646668 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:29.647307 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 07:22:29.647511 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:22:29.648598 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.635206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:57791)
0926 07:22:29.635970 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:29.636026 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:29.637009 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:29.637555 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:29.637563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:29.637990 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:29.638479 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.638494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.639266 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.639269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.640158 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.640167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.640841 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.640844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.640994 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.641680 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:29.641697 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:29.643253 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:29.645448 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:29.645453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:29.645468 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:29.645765 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:29.646106 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:29.646113 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:29.646115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:29.646298 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:29.646728 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:29.646734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:29.647036 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 07:22:29.647213 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 07:22:29.647550 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
I20260926 07:22:29.649935 12871 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43069
I20260926 07:22:29.650198 12871 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:29.650825 12871 heartbeater.cc:511] Master 127.28.235.254:43069 requested a full tablet report, sending...
I20260926 07:22:29.652130 12546 ts_manager.cc:194] Registered new tserver with Master: a61de86d26384c678c85445894340a2d (127.28.235.194:44013)
I20260926 07:22:29.652974 12546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:57791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:22:29.668308 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43069
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:29.790117 12876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:29.790413 12876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:29.790458 12876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:29.794878 12876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:29.794956 12876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:29.795043 12876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:22:29.800700 12876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:29.801888 12876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:29.802987 12876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:29.813520 12881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:29.813521 12884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:29.813521 12882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:29.813696 12876 server_base.cc:1034] running on GCE node
I20260926 07:22:29.814373 12876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:29.815169 12876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:29.816381 12876 hybrid_clock.cc:648] HybridClock initialized: now 1790407349816350 us; error 33 us; skew 500 ppm
Sep 26 07:22:29 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407349, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:29.819550 12876 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:22:29.820611 12876 webserver.cc:460] Webserver started at http://127.28.235.195:42849/ using document root <none> and password file <none>
I20260926 07:22:29.821285 12876 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:29.821377 12876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:29.821612 12876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:22:29.823801 12876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "4784fb4e343342f5881e2135fad697ef"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "56746f5066244ac8ccca976223619b69"
server_key_iv: "d8cf590b4d6f844b73a568e0bca41e72"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.824427 12876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "4784fb4e343342f5881e2135fad697ef"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "56746f5066244ac8ccca976223619b69"
server_key_iv: "d8cf590b4d6f844b73a568e0bca41e72"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.829331 12876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 07:22:29.832324 12891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:29.833885 12876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:29.834044 12876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "4784fb4e343342f5881e2135fad697ef"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "56746f5066244ac8ccca976223619b69"
server_key_iv: "d8cf590b4d6f844b73a568e0bca41e72"
server_key_version: "encryptionkey@0"
I20260926 07:22:29.834179 12876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:29.858346 12876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:29.861795 12876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:29.862004 12876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:29.862620 12876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:29.863511 12876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:29.863580 12876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:29.863648 12876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:29.863695 12876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:29.873903 12876 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:46147
I20260926 07:22:29.873955 13004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:46147 every 8 connection(s)
I20260926 07:22:29.874899 12876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:22:29.876312 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 12876
I20260926 07:22:29.876432 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:22:29.876771 29615 external_mini_cluster.cc:1428] Setting key 7c5e457a4c0e60e2e6e0bd48094bb143
Sep 26 07:22:29 dist-test-slave-7zlw krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407349, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:29.889611 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.876847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:34487 (local address 127.28.235.254:43069)
0926 07:22:29.876981 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:29.876985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:29.878384 (+ 1399us) server_negotiation.cc:407] Connection header received
0926 07:22:29.879293 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:29.879296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:29.879342 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:29.879422 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:29.880792 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.881436 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.882797 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.883008 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.883091 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.885941 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:29.885946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:29.885953 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:29.885981 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:29.887325 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:29.887861 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:29.887865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:29.887867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:29.887947 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:29.888306 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:29.888311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:29.888312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:29.888507 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:29.888634 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:29.889227 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 07:22:29.889404 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 07:22:29.890610 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.877480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:34487)
0926 07:22:29.878215 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:29.878281 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:29.879059 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:29.879683 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:29.879691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:29.880097 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:29.880630 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.880645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.881595 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.881599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.882653 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.882665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.883165 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.883168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.883363 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.884020 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:29.884037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:29.885748 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:29.887457 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:29.887463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:29.887482 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:29.887757 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:29.888039 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:29.888044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:29.888046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:29.888195 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:29.888633 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:29.888641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:29.888908 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 07:22:29.889148 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 07:22:29.889487 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":151,"threads_started":1}
I20260926 07:22:29.892019 13005 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43069
I20260926 07:22:29.892330 13005 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:29.892926 13005 heartbeater.cc:511] Master 127.28.235.254:43069 requested a full tablet report, sending...
I20260926 07:22:29.894268 12546 ts_manager.cc:194] Registered new tserver with Master: 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147)
I20260926 07:22:29.895021 12546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:34487
I20260926 07:22:29.903231 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:22:29.914347 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.905615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.28.235.254:43069)
0926 07:22:29.905775 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:29.905781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:29.905962 (+ 181us) server_negotiation.cc:407] Connection header received
0926 07:22:29.906061 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:29.906064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:29.906109 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:29.906193 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:29.907144 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.907715 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.908655 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.908918 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.909079 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.910642 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:29.910646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:29.910648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:29.910675 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:29.912296 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:29.912830 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:29.912834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:29.912835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:29.912931 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:29.913293 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:29.913296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:29.913297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:29.913549 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:29.913655 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:29.913950 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 07:22:29.914064 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:22:29.917752 12546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:22:29.921587 12546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:22:29.936590 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.931642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.28.235.193:38343)
0926 07:22:29.932276 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:29.932283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:29.932307 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:22:29.932387 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:29.932391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:29.932559 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:29.932715 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:29.932993 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.933827 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.935332 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.935884 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.936021 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.936172 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:29.936327 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 07:22:29.936428 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":93,"threads_started":1}
I20260926 07:22:29.936581 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.931031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38343 (local address 127.0.0.1:49702)
0926 07:22:29.931815 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:29.931829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:29.931924 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:29.932705 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:29.932720 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:29.932742 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:29.932878 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.932885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.933950 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.933956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.935149 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.935169 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.936008 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.936012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.936121 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.936159 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:22:29.936296 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 07:22:29.936428 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":100,"threads_started":1}
I20260926 07:22:29.937098 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.930834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:44013 (local address 127.0.0.1:51018)
0926 07:22:29.931679 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:29.931735 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:29.931873 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:29.932859 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:29.932862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:29.932879 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:29.933073 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.933084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.934578 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.934582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.935578 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.935588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.936652 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.936668 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.936755 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.936781 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:22:29.936829 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:22:29.936913 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":78,"thread_start_us":98,"threads_started":1}
I20260926 07:22:29.937845 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.931031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51018 (local address 127.28.235.194:44013)
0926 07:22:29.932446 (+ 1415us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:29.932457 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:29.932476 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:22:29.932540 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:29.932544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:29.932711 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:29.933102 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:29.933279 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.934418 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.935712 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.936516 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.937402 (+ 886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.937536 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:29.937588 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 07:22:29.937701 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1313,"thread_start_us":107,"threads_started":1}
I20260926 07:22:29.938647 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.931289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:46147 (local address 127.0.0.1:35240)
0926 07:22:29.932163 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:29.932177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:29.932264 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:29.932931 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:29.932934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:29.932946 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:29.933076 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.933083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.934578 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.934582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.936815 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:29.936825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.938260 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.938263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:29.938338 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.938356 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:22:29.938448 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:22:29.938522 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":10,"thread_start_us":97,"threads_started":1}
I20260926 07:22:29.938747 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:29.931673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.28.235.195:46147)
0926 07:22:29.932191 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:29.932207 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:29.932235 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:22:29.932417 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:29.932422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:29.932615 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:29.932768 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:29.933220 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.934418 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.936943 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:29.937842 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:29.938023 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:29.938116 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:29.938493 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 07:22:29.938588 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":102,"threads_started":1}
I20260926 07:22:29.940418 12671 tablet_service.cc:1467] Processing CreateTablet for tablet 9d85427e02a74e28ad2945c1d56749a1 (DEFAULT_TABLE table=test-table [id=c0d4740851fe4de58c3ef1a14409d1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:29.940415 12805 tablet_service.cc:1467] Processing CreateTablet for tablet 9d85427e02a74e28ad2945c1d56749a1 (DEFAULT_TABLE table=test-table [id=c0d4740851fe4de58c3ef1a14409d1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:29.941668 12805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d85427e02a74e28ad2945c1d56749a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:29.941668 12671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d85427e02a74e28ad2945c1d56749a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:29.941466 12939 tablet_service.cc:1467] Processing CreateTablet for tablet 9d85427e02a74e28ad2945c1d56749a1 (DEFAULT_TABLE table=test-table [id=c0d4740851fe4de58c3ef1a14409d1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:29.942416 12939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d85427e02a74e28ad2945c1d56749a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:29.950040 13023 tablet_bootstrap.cc:492] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Bootstrap starting.
I20260926 07:22:29.950460 13022 tablet_bootstrap.cc:492] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: Bootstrap starting.
I20260926 07:22:29.950491 13024 tablet_bootstrap.cc:492] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: Bootstrap starting.
I20260926 07:22:29.952425 13022 tablet_bootstrap.cc:654] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:29.952864 13024 tablet_bootstrap.cc:654] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:29.953621 13022 log.cc:824] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:29.953760 13024 log.cc:824] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:29.955471 13023 tablet_bootstrap.cc:654] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:29.955929 13024 tablet_bootstrap.cc:492] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: No bootstrap required, opened a new log
I20260926 07:22:29.955943 13022 tablet_bootstrap.cc:492] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: No bootstrap required, opened a new log
I20260926 07:22:29.956151 13024 ts_tablet_manager.cc:1397] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:22:29.956153 13022 ts_tablet_manager.cc:1397] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 07:22:29.956758 13023 log.cc:824] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:29.959196 13023 tablet_bootstrap.cc:492] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: No bootstrap required, opened a new log
I20260926 07:22:29.959420 13023 ts_tablet_manager.cc:1397] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260926 07:22:29.959676 13024 raft_consensus.cc:348] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:29.959944 13024 raft_consensus.cc:374] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:29.960017 13024 raft_consensus.cc:729] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61de86d26384c678c85445894340a2d, State: Initialized, Role: FOLLOWER
I20260926 07:22:29.959975 13022 raft_consensus.cc:348] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:29.960283 13022 raft_consensus.cc:374] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:29.960337 13022 raft_consensus.cc:729] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4784fb4e343342f5881e2135fad697ef, State: Initialized, Role: FOLLOWER
I20260926 07:22:29.960542 13024 consensus_queue.cc:260] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:29.960803 13022 consensus_queue.cc:260] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:29.961314 12871 heartbeater.cc:503] Master 127.28.235.254:43069 was elected leader, sending a full tablet report...
I20260926 07:22:29.961643 13024 ts_tablet_manager.cc:1428] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 07:22:29.961932 13022 ts_tablet_manager.cc:1428] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:22:29.962285 13005 heartbeater.cc:503] Master 127.28.235.254:43069 was elected leader, sending a full tablet report...
I20260926 07:22:29.965070 13023 raft_consensus.cc:348] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:29.965428 13023 raft_consensus.cc:374] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:29.965533 13023 raft_consensus.cc:729] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ca4960845f43f8b0c66d5ca6074c84, State: Initialized, Role: FOLLOWER
I20260926 07:22:29.966038 13023 consensus_queue.cc:260] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:29.966831 12737 heartbeater.cc:503] Master 127.28.235.254:43069 was elected leader, sending a full tablet report...
I20260926 07:22:29.967162 13023 ts_tablet_manager.cc:1428] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260926 07:22:30.021340 13028 raft_consensus.cc:482] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:22:30.021617 13028 raft_consensus.cc:504] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:30.022961 13028 leader_election.cc:288] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1ca4960845f43f8b0c66d5ca6074c84 (127.28.235.193:38343), 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147)
I20260926 07:22:30.028199 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:30.023482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:60457 (local address 127.28.235.193:38343)
0926 07:22:30.023617 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:30.023621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:30.024062 (+ 441us) server_negotiation.cc:407] Connection header received
0926 07:22:30.024172 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:30.024174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:30.024213 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:30.024280 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:30.024536 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:30.025315 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:30.026905 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:30.027514 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:30.027642 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:30.027697 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:30.027982 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 07:22:30.028056 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:22:30.028199 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:30.023451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:47381 (local address 127.28.235.195:46147)
0926 07:22:30.023582 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:30.023587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:30.023609 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:22:30.023852 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:30.023854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:30.023901 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:30.023961 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:30.024345 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:30.025134 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:30.026838 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:30.027514 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:30.027642 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:30.027697 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:30.027981 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 07:22:30.028056 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:22:30.028235 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:30.023274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:46147 (local address 127.28.235.194:47381)
0926 07:22:30.023479 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:30.023497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:30.023643 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:30.024023 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:30.024028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:30.024051 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:30.024214 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:30.024220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:30.025260 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:30.025263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:30.026707 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:30.026720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:30.027656 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:30.027661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:30.027812 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:30.027861 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 07:22:30.027954 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:22:30.028063 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":33}
I20260926 07:22:30.028245 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:30.023274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38343 (local address 127.28.235.194:60457)
0926 07:22:30.023926 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:30.023963 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:30.024070 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:30.024311 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:30.024314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:30.024336 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:30.024435 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:30.024440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:30.025416 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:30.025419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:30.026790 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:30.026803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:30.027656 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:30.027661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:30.027828 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:30.027880 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:22:30.027954 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:22:30.028055 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":327,"spinlock_wait_cycles":3456,"thread_start_us":128,"threads_started":1}
I20260926 07:22:30.029073 12959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d85427e02a74e28ad2945c1d56749a1" candidate_uuid: "a61de86d26384c678c85445894340a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4784fb4e343342f5881e2135fad697ef" is_pre_election: true
I20260926 07:22:30.029207 12691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d85427e02a74e28ad2945c1d56749a1" candidate_uuid: "a61de86d26384c678c85445894340a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" is_pre_election: true
I20260926 07:22:30.029456 12959 raft_consensus.cc:2454] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a61de86d26384c678c85445894340a2d in term 0.
I20260926 07:22:30.029510 12691 raft_consensus.cc:2454] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a61de86d26384c678c85445894340a2d in term 0.
I20260926 07:22:30.030097 12758 leader_election.cc:302] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4784fb4e343342f5881e2135fad697ef, a61de86d26384c678c85445894340a2d, f1ca4960845f43f8b0c66d5ca6074c84; no voters:
I20260926 07:22:30.030432 13028 raft_consensus.cc:2781] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:22:30.030527 13028 raft_consensus.cc:482] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:22:30.030584 13028 raft_consensus.cc:3037] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:30.031729 13028 raft_consensus.cc:504] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:30.032114 13028 leader_election.cc:288] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [CANDIDATE]: Term 1 election: Requested vote from peers f1ca4960845f43f8b0c66d5ca6074c84 (127.28.235.193:38343), 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147)
I20260926 07:22:30.032449 12691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d85427e02a74e28ad2945c1d56749a1" candidate_uuid: "a61de86d26384c678c85445894340a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ca4960845f43f8b0c66d5ca6074c84"
I20260926 07:22:30.032480 12959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d85427e02a74e28ad2945c1d56749a1" candidate_uuid: "a61de86d26384c678c85445894340a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4784fb4e343342f5881e2135fad697ef"
I20260926 07:22:30.032595 12691 raft_consensus.cc:3037] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:30.032594 12959 raft_consensus.cc:3037] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:30.034041 12959 raft_consensus.cc:2454] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a61de86d26384c678c85445894340a2d in term 1.
I20260926 07:22:30.034041 12691 raft_consensus.cc:2454] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a61de86d26384c678c85445894340a2d in term 1.
I20260926 07:22:30.034420 12759 leader_election.cc:302] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4784fb4e343342f5881e2135fad697ef, a61de86d26384c678c85445894340a2d, f1ca4960845f43f8b0c66d5ca6074c84; no voters:
I20260926 07:22:30.034653 13028 raft_consensus.cc:2781] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:22:30.034863 13028 raft_consensus.cc:686] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 1 LEADER]: Becoming Leader. State: Replica: a61de86d26384c678c85445894340a2d, State: Running, Role: LEADER
I20260926 07:22:30.035229 13028 consensus_queue.cc:237] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:30.038901 12546 catalog_manager.cc:5495] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d reported cstate change: term changed from 0 to 1, leader changed from <none> to a61de86d26384c678c85445894340a2d (127.28.235.194). New cstate: current_term: 1 leader_uuid: "a61de86d26384c678c85445894340a2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } health_report { overall_health: UNKNOWN } } }
I20260926 07:22:30.064633 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:30.061144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.28.235.194:44013)
0926 07:22:30.061287 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:30.061291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:30.061305 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:22:30.061392 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:30.061394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:30.061451 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:30.061550 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:22:30.061781 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:30.062313 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:30.063177 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:30.063402 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:30.063487 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:30.063784 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:22:30.064235 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:22:30.064327 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:30.064426 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:22:30.064478 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:22:30.071116 12959 raft_consensus.cc:1261] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef [term 1 FOLLOWER]: Refusing update from remote peer a61de86d26384c678c85445894340a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:30.071116 12691 raft_consensus.cc:1261] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Refusing update from remote peer a61de86d26384c678c85445894340a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:30.071868 13028 consensus_queue.cc:1035] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:30.072090 13032 consensus_queue.cc:1035] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:30.081727 13037 mvcc.cc:205] Tried to move back new op lower bound from 7333508505883197440 to 7333508505747193856. Current Snapshot: MvccSnapshot[applied={T|T < 7333508505883197440}]
I20260926 07:22:30.090669 12545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:22:30.090945 12545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:22:30.093946 12545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Sending DeleteTablet for 3 replicas of tablet 9d85427e02a74e28ad2945c1d56749a1
I20260926 07:22:30.095007 12805 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d85427e02a74e28ad2945c1d56749a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:30 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:51018
I20260926 07:22:30.095007 12671 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d85427e02a74e28ad2945c1d56749a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:30 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:49702
I20260926 07:22:30.095603 13045 tablet_replica.cc:330] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: stopping tablet replica
I20260926 07:22:30.095611 12939 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d85427e02a74e28ad2945c1d56749a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:30 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:35240
I20260926 07:22:30.095852 13046 tablet_replica.cc:330] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: stopping tablet replica
I20260926 07:22:30.096131 13047 tablet_replica.cc:330] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: stopping tablet replica
I20260926 07:22:30.096220 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 12608
I20260926 07:22:30.096218 13046 raft_consensus.cc:2229] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:22:30.096513 13047 raft_consensus.cc:2229] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:22:30.096676 13046 raft_consensus.cc:2258] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:30.096832 13047 raft_consensus.cc:2258] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:30.098404 13046 ts_tablet_manager.cc:1905] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:30.098536 13047 ts_tablet_manager.cc:1905] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:30.101441 13046 ts_tablet_manager.cc:1918] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:22:30.101640 13046 log.cc:1196] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/9d85427e02a74e28ad2945c1d56749a1
I20260926 07:22:30.102023 13046 ts_tablet_manager.cc:1939] T 9d85427e02a74e28ad2945c1d56749a1 P a61de86d26384c678c85445894340a2d: Deleting consensus metadata
I20260926 07:22:30.102304 13047 ts_tablet_manager.cc:1918] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:22:30.102485 13047 log.cc:1196] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/9d85427e02a74e28ad2945c1d56749a1
I20260926 07:22:30.102878 13047 ts_tablet_manager.cc:1939] T 9d85427e02a74e28ad2945c1d56749a1 P 4784fb4e343342f5881e2135fad697ef: Deleting consensus metadata
I20260926 07:22:30.103448 12533 catalog_manager.cc:4841] TS a61de86d26384c678c85445894340a2d (127.28.235.194:44013): tablet 9d85427e02a74e28ad2945c1d56749a1 (table test-table [id=c0d4740851fe4de58c3ef1a14409d1f3]) successfully deleted
I20260926 07:22:30.104246 12530 catalog_manager.cc:4841] TS 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147): tablet 9d85427e02a74e28ad2945c1d56749a1 (table test-table [id=c0d4740851fe4de58c3ef1a14409d1f3]) successfully deleted
W20260926 07:22:30.105928 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:38343 (error 108)
I20260926 07:22:30.106858 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 12742
I20260926 07:22:30.107110 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:30.106796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38343 (local address 127.0.0.1:49716)
0926 07:22:30.106935 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:30.107009 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:38343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 07:22:30.111989 12531 catalog_manager.cc:4584] TS f1ca4960845f43f8b0c66d5ca6074c84 (127.28.235.193:38343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d85427e02a74e28ad2945c1d56749a1: Network error: Client connection negotiation failed: client connection to 127.28.235.193:38343: connect: Connection refused (error 111)
I20260926 07:22:30.113130 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 12876
I20260926 07:22:30.118924 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 12514
I20260926 07:22:50.126188 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43069
--webserver_interface=127.28.235.254
--webserver_port=35625
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:22:50.246003 13049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:50.246311 13049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:50.246356 13049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:50.250542 13049 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:22:50.250617 13049 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:22:50.250635 13049 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:22:50.250649 13049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:22:50.250670 13049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:50.250695 13049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:50.250712 13049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:22:50.250728 13049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:22:50.256527 13049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:43069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:50.257663 13049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:50.258699 13049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:50.273756 13056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:50.273834 13055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:50.273891 13058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:50.274371 13049 server_base.cc:1034] running on GCE node
I20260926 07:22:50.275056 13049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:50.276404 13049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:50.277624 13049 hybrid_clock.cc:648] HybridClock initialized: now 1790407370277591 us; error 53 us; skew 500 ppm
Sep 26 07:22:50 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407370, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:50.280756 13049 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:22:50.281807 13049 webserver.cc:460] Webserver started at http://127.28.235.254:35625/ using document root <none> and password file <none>
I20260926 07:22:50.282413 13049 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:50.282488 13049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:50.286841 13049 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 07:22:50.289880 13065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:50.291543 13049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:50.291826 13049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "2c536ffc3b9744aeafa34fd4c16c96f9"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "adb15f9f33f67541e4bfd68288823cb3"
server_key_iv: "7b6fe4491861fba90000500b84e5f062"
server_key_version: "encryptionkey@0"
I20260926 07:22:50.292591 13049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:50.315464 13049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:50.319037 13049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:50.319265 13049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:50.327631 13049 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:43069
I20260926 07:22:50.328037 13117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:43069 every 8 connection(s)
I20260926 07:22:50.328850 13049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:22:50.334471 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 13049
I20260926 07:22:50.334990 13118 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:22:50.335021 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:38343
--local_ip_for_outbound_sockets=127.28.235.193
--tserver_master_addrs=127.28.235.254:43069
--webserver_port=42631
--webserver_interface=127.28.235.193
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:22:50.337435 13118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Bootstrap starting.
I20260926 07:22:50.359015 13118 log.cc:824] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:50.371084 13118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:22:50.371623 13118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Bootstrap complete.
I20260926 07:22:50.376927 13118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:22:50.377674 13118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c536ffc3b9744aeafa34fd4c16c96f9, State: Initialized, Role: FOLLOWER
I20260926 07:22:50.378366 13118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:22:50.378556 13118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:22:50.378671 13118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:22:50.378818 13118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:22:50.381671 13118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:22:50.382159 13118 leader_election.cc:302] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c536ffc3b9744aeafa34fd4c16c96f9; no voters:
I20260926 07:22:50.382623 13118 leader_election.cc:288] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:22:50.382752 13122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:22:50.383025 13122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 2 LEADER]: Becoming Leader. State: Replica: 2c536ffc3b9744aeafa34fd4c16c96f9, State: Running, Role: LEADER
I20260926 07:22:50.383517 13122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:22:50.384397 13118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:22:50.385404 13123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } } }
I20260926 07:22:50.385574 13123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:50.386497 13124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c536ffc3b9744aeafa34fd4c16c96f9. Latest consensus state: current_term: 2 leader_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } } }
I20260926 07:22:50.386741 13124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:22:50.387168 13129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:22:50.392688 13129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:22:50.393472 13129 catalog_manager.cc:1259] Loaded cluster ID: ce42ce341f98431bbb26f34b4cac36d6
I20260926 07:22:50.393563 13129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:22:50.395896 13129 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:22:50.396749 13129 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Loaded TSK: 0
I20260926 07:22:50.414413 13129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Generated new TSK 1
I20260926 07:22:50.414841 13129 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:22:50.479853 13120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:50.480269 13120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:50.480360 13120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:50.486523 13120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:50.486622 13120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:50.486712 13120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:22:50.494640 13120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:38343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=42631
--tserver_master_addrs=127.28.235.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:50.496074 13120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:50.497457 13120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:50.510354 13148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:50.510313 13145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:50.510301 13146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:50.510814 13120 server_base.cc:1034] running on GCE node
I20260926 07:22:50.511341 13120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:50.512213 13120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:50.513461 13120 hybrid_clock.cc:648] HybridClock initialized: now 1790407370513438 us; error 45 us; skew 500 ppm
Sep 26 07:22:50 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407370, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:50.516796 13120 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:22:50.518042 13120 webserver.cc:460] Webserver started at http://127.28.235.193:42631/ using document root <none> and password file <none>
I20260926 07:22:50.518749 13120 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:50.518831 13120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:50.523977 13120 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:22:50.526758 13155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:50.528278 13120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:22:50.528441 13120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "f1ca4960845f43f8b0c66d5ca6074c84"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "fe500d24e310c732ed99a38ce8f49a91"
server_key_iv: "fc876c52eaae92c3c731c57d5c4ef456"
server_key_version: "encryptionkey@0"
I20260926 07:22:50.529177 13120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:50.545727 13120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:50.550347 13120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:50.550617 13120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:50.551380 13120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:50.552865 13162 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:22:50.558883 13120 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:22:50.559055 13120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 07:22:50.559170 13120 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:22:50.561724 13120 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:22:50.561820 13120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260926 07:22:50.562299 13162 tablet_bootstrap.cc:492] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Bootstrap starting.
I20260926 07:22:50.574489 13120 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:38343
I20260926 07:22:50.574760 13269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:38343 every 8 connection(s)
I20260926 07:22:50.575770 13120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:22:50.583258 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 13120
I20260926 07:22:50.583819 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:44013
--local_ip_for_outbound_sockets=127.28.235.194
--tserver_master_addrs=127.28.235.254:43069
--webserver_port=45703
--webserver_interface=127.28.235.194
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:22:50.585654 13162 log.cc:824] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:22:50 dist-test-slave-7zlw krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407370, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:50.596364 13162 tablet_bootstrap.cc:492] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:22:50.597013 13162 tablet_bootstrap.cc:492] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Bootstrap complete.
I20260926 07:22:50.598583 13162 ts_tablet_manager.cc:1397] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.007s
I20260926 07:22:50.599397 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:50.578344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:48745 (local address 127.28.235.254:43069)
0926 07:22:50.579026 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:50.579051 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:50.580617 (+ 1566us) server_negotiation.cc:407] Connection header received
0926 07:22:50.580888 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:50.580898 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:50.581327 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:50.581744 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:50.583151 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.584793 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.586785 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.587594 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.587788 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:50.591846 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:50.591858 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:50.591888 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:50.591943 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:50.594626 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:50.595450 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:50.595459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:50.595469 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:50.595671 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:50.596135 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:50.596141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:50.596144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:50.596801 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:50.596977 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:50.597770 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 07:22:50.598161 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":153,"threads_started":1}
I20260926 07:22:50.599534 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:50.578386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:48745)
0926 07:22:50.579355 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:50.579418 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:50.580351 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:50.581804 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:50.581813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:50.582321 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:50.582909 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:50.582939 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.585073 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.585080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:50.586636 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:50.586648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.587817 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.587827 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:50.588153 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:50.589120 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:50.589150 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:50.591510 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:50.594825 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:50.594832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:50.594847 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:50.595283 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:50.595811 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:50.595814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:50.595816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:50.596011 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:50.596951 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:50.596958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:50.597354 (+ 396us) client_negotiation.cc:769] Sending connection context
0926 07:22:50.597589 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 07:22:50.598226 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":180,"threads_started":1}
I20260926 07:22:50.602090 13162 raft_consensus.cc:348] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:50.602662 13162 raft_consensus.cc:729] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ca4960845f43f8b0c66d5ca6074c84, State: Initialized, Role: FOLLOWER
I20260926 07:22:50.603133 13270 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43069
I20260926 07:22:50.603402 13270 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:50.603312 13162 consensus_queue.cc:260] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } }
I20260926 07:22:50.604185 13270 heartbeater.cc:511] Master 127.28.235.254:43069 requested a full tablet report, sending...
I20260926 07:22:50.604621 13162 ts_tablet_manager.cc:1428] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260926 07:22:50.607824 13082 ts_manager.cc:194] Registered new tserver with Master: f1ca4960845f43f8b0c66d5ca6074c84 (127.28.235.193:38343)
I20260926 07:22:50.612499 13082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:48745
I20260926 07:22:50.615931 13270 heartbeater.cc:503] Master 127.28.235.254:43069 was elected leader, sending a full tablet report...
Sep 26 07:22:50 dist-test-slave-7zlw krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407370, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.193@KRBTEST.COM
I20260926 07:22:50.625083 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:50.612091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38343 (local address 127.0.0.1:59150)
0926 07:22:50.613284 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:50.613330 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:50.613552 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:50.614241 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:50.614248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:50.614728 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:50.614883 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:50.614893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.616249 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.616253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:50.617264 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:50.617276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.617938 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.617944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:50.618071 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:50.618838 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:50.618852 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:50.620735 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:50.622999 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:50.623011 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:50.623018 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:50.623275 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:50.623619 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:50.623622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:50.623624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:50.623714 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:50.624253 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:50.624261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:50.624413 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 07:22:50.624576 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 07:22:50.624833 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":197,"threads_started":1}
I20260926 07:22:50.625083 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:50.612162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59150 (local address 127.28.235.193:38343)
0926 07:22:50.612941 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:50.612955 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:50.613603 (+ 648us) server_negotiation.cc:407] Connection header received
0926 07:22:50.613750 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:50.613755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:50.614024 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:50.614184 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:50.615062 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.616090 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.617402 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.617795 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.618072 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:50.621061 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:50.621074 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:50.621097 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:50.621122 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:50.622866 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:50.623412 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:50.623420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:50.623427 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:50.623504 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:50.623819 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:50.623822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:50.623824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:50.624126 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:50.624284 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:50.624662 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 07:22:50.624852 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":227,"threads_started":1}
I20260926 07:22:50.626499 13204 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d85427e02a74e28ad2945c1d56749a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:30 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:59150
I20260926 07:22:50.627532 13279 tablet_replica.cc:330] stopping tablet replica
I20260926 07:22:50.627911 13279 raft_consensus.cc:2229] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:22:50.628234 13279 raft_consensus.cc:2258] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:50.629563 13279 ts_tablet_manager.cc:1905] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:50.634197 13279 ts_tablet_manager.cc:1918] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:22:50.634349 13279 log.cc:1196] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal/wals/9d85427e02a74e28ad2945c1d56749a1
I20260926 07:22:50.634670 13279 ts_tablet_manager.cc:1939] T 9d85427e02a74e28ad2945c1d56749a1 P f1ca4960845f43f8b0c66d5ca6074c84: Deleting consensus metadata
I20260926 07:22:50.636912 13067 catalog_manager.cc:4841] TS f1ca4960845f43f8b0c66d5ca6074c84 (127.28.235.193:38343): tablet 9d85427e02a74e28ad2945c1d56749a1 (table test-table [id=c0d4740851fe4de58c3ef1a14409d1f3]) successfully deleted
W20260926 07:22:50.709396 13274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:50.709692 13274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:50.709735 13274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:50.713867 13274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:50.713946 13274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:50.714015 13274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:22:50.719444 13274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:44013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=45703
--tserver_master_addrs=127.28.235.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:50.720568 13274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:50.721740 13274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:50.732911 13284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:50.732921 13285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:50.732963 13287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:50.733135 13274 server_base.cc:1034] running on GCE node
I20260926 07:22:50.733975 13274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:50.734773 13274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:50.736012 13274 hybrid_clock.cc:648] HybridClock initialized: now 1790407370735960 us; error 48 us; skew 500 ppm
Sep 26 07:22:50 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407370, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:50.739199 13274 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:22:50.740252 13274 webserver.cc:460] Webserver started at http://127.28.235.194:45703/ using document root <none> and password file <none>
I20260926 07:22:50.740970 13274 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:50.741153 13274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:50.745906 13274 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 07:22:50.748564 13294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:50.750376 13274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:22:50.750559 13274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "a61de86d26384c678c85445894340a2d"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "7688e30ef3a20d7a355ee8b3d7ce08fa"
server_key_iv: "5617fe176e035f1b7f64a8be58f4e9d5"
server_key_version: "encryptionkey@0"
I20260926 07:22:50.751173 13274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:50.768868 13274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:50.774574 13274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:50.774901 13274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:50.775805 13274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:50.777491 13274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:50.777552 13274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:50.777632 13274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:50.777667 13274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:50.790561 13274 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:44013
I20260926 07:22:50.790653 13407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:44013 every 8 connection(s)
I20260926 07:22:50.791945 13274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:22:50.793416 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 13274
I20260926 07:22:50.793921 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:46147
--local_ip_for_outbound_sockets=127.28.235.195
--tserver_master_addrs=127.28.235.254:43069
--webserver_port=42849
--webserver_interface=127.28.235.195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:22:50 dist-test-slave-7zlw krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407370, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:50.807829 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:50.796099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:32915 (local address 127.28.235.254:43069)
0926 07:22:50.796295 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:50.796301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:50.796320 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:22:50.796916 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:50.796919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:50.796981 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:50.797093 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:50.798489 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.799054 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.800202 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.800452 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.800569 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:50.803191 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:50.803195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:50.803197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:50.803220 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:50.805458 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:50.806181 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:50.806187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:50.806189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:50.806270 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:50.806627 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:50.806630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:50.806632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:50.806826 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:50.806934 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:50.807475 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 07:22:50.807636 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 07:22:50.808738 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:50.794783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:32915)
0926 07:22:50.795607 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:50.795663 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:50.796712 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:50.797405 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:50.797426 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:50.797842 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:50.798341 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:50.798354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.799185 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.799189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:50.800074 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:50.800084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:50.800625 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:50.800629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:50.800796 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:50.801446 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:50.801463 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:50.802995 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:50.805673 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:50.805680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:50.805702 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:50.806060 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:50.806375 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:50.806378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:50.806380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:50.806522 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:50.806933 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:50.806941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:50.807237 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 07:22:50.807437 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 07:22:50.807783 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":174,"threads_started":1}
I20260926 07:22:50.810081 13408 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43069
I20260926 07:22:50.810508 13408 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:50.811301 13408 heartbeater.cc:511] Master 127.28.235.254:43069 requested a full tablet report, sending...
I20260926 07:22:50.812742 13082 ts_manager.cc:194] Registered new tserver with Master: a61de86d26384c678c85445894340a2d (127.28.235.194:44013)
I20260926 07:22:50.813434 13082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:32915
W20260926 07:22:50.933339 13411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:22:50.933738 13411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:22:50.933815 13411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:22:50.938336 13411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:22:50.938429 13411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:22:50.938531 13411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:22:50.944680 13411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:46147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=42849
--tserver_master_addrs=127.28.235.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:22:50.946324 13411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:22:50.947690 13411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:22:50.960716 13417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:50.960706 13416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:22:50.960741 13419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:22:50.961367 13411 server_base.cc:1034] running on GCE node
I20260926 07:22:50.962009 13411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:22:50.962945 13411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:22:50.964161 13411 hybrid_clock.cc:648] HybridClock initialized: now 1790407370964126 us; error 53 us; skew 500 ppm
Sep 26 07:22:50 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407370, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:22:50.967739 13411 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:22:50.968954 13411 webserver.cc:460] Webserver started at http://127.28.235.195:42849/ using document root <none> and password file <none>
I20260926 07:22:50.969748 13411 fs_manager.cc:359] Metadata directory not provided
I20260926 07:22:50.969882 13411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:22:50.974773 13411 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:22:50.977532 13426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:50.979241 13411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:22:50.979446 13411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "4784fb4e343342f5881e2135fad697ef"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "56746f5066244ac8ccca976223619b69"
server_key_iv: "d8cf590b4d6f844b73a568e0bca41e72"
server_key_version: "encryptionkey@0"
I20260926 07:22:50.980280 13411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:22:50.993512 13411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:22:50.997932 13411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:22:50.998181 13411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:22:50.999189 13411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:22:51.000687 13411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:22:51.000757 13411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:51.000821 13411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:22:51.000851 13411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:22:51.014356 13411 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:46147
I20260926 07:22:51.014387 13539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:46147 every 8 connection(s)
I20260926 07:22:51.015657 13411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:22:51.023125 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 13411
Sep 26 07:22:51 dist-test-slave-7zlw krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407370, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:22:51.038564 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.024109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.28.235.254:43069)
0926 07:22:51.024443 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:51.024447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:51.024569 (+ 122us) server_negotiation.cc:407] Connection header received
0926 07:22:51.024840 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:51.024844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:51.024909 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:51.025104 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:51.025950 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.026649 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.027639 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.027892 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.028004 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.033808 (+ 5804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:51.033812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:51.033852 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:51.033889 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:51.036341 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:51.036943 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:51.036949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:51.036954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:51.037084 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:51.037444 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:51.037449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:51.037451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:51.037734 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:51.038022 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:51.038150 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 07:22:51.038329 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260926 07:22:51.039548 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.018210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:58701 (local address 127.28.235.254:43069)
0926 07:22:51.018394 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:51.018399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:51.019772 (+ 1373us) server_negotiation.cc:407] Connection header received
0926 07:22:51.020819 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:51.020823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:51.020883 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:51.020988 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:51.022441 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.023003 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.024874 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.025217 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.025337 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.035060 (+ 9723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:51.035064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:51.035069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:51.035125 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:51.036909 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:51.037555 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:51.037560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:51.037562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:51.037675 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:51.038050 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:51.038054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:51.038056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:51.038297 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:51.038411 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:51.039209 (+ 798us) server_negotiation.cc:299] Negotiation successful
0926 07:22:51.039347 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":79}
I20260926 07:22:51.040443 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.018704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:58701)
0926 07:22:51.019582 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:51.019638 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:51.020575 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:51.021233 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:51.021241 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:51.021662 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:22:51.022289 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:51.022303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.023131 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.023134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:51.024726 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:51.024738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.025399 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.025403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:51.025584 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.032661 (+ 7077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:22:51.032682 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:22:51.034731 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:22:51.037093 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:51.037100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:51.037116 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:51.037409 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:51.037782 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:22:51.037787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:22:51.037789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:22:51.037938 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:22:51.038406 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:22:51.038414 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:22:51.038724 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 07:22:51.038962 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 07:22:51.039356 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":145,"threads_started":1}
I20260926 07:22:51.041930 13540 heartbeater.cc:347] Connected to a master server at 127.28.235.254:43069
I20260926 07:22:51.042179 13540 heartbeater.cc:464] Registering TS with master...
I20260926 07:22:51.042704 13540 heartbeater.cc:511] Master 127.28.235.254:43069 requested a full tablet report, sending...
I20260926 07:22:51.043783 13082 ts_manager.cc:194] Registered new tserver with Master: 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147)
I20260926 07:22:51.044370 13082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:58701
I20260926 07:22:51.053959 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:22:51.059350 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.054992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.28.235.254:43069)
0926 07:22:51.055158 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:51.055162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:51.055231 (+ 69us) server_negotiation.cc:407] Connection header received
0926 07:22:51.055453 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:51.055456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:51.055608 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:51.055732 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:22:51.056194 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.056773 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.057918 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.058150 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.058246 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.058670 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:22:51.058869 (+ 199us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 07:22:51.059139 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42040: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 07:22:51.059487 13272 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42040: token expired
W20260926 07:22:51.059861 13009 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.235.254:43069)
I20260926 07:22:51.068306 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.060347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.28.235.254:43069)
0926 07:22:51.060497 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:51.060500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:51.060511 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:22:51.060544 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:51.060546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:51.060605 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:51.060678 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:22:51.061403 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.061911 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.062787 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.063004 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.063086 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.064543 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:22:51.064546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:22:51.064549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:22:51.064568 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:22:51.066395 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:51.066976 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:51.066979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:51.066981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:51.067070 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:22:51.067421 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:22:51.067424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:22:51.067425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:22:51.067641 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:22:51.067774 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:51.068032 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 07:22:51.068150 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 07:22:51.072088 13082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:22:51.076385 13082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:22:51.087589 13204 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4ae879036247dea7671b838084aeee (DEFAULT_TABLE table=test-table [id=f42aa580c12b493aaf9dca48a138c800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:51.088510 13204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4ae879036247dea7671b838084aeee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:51.092208 13162 tablet_bootstrap.cc:492] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: Bootstrap starting.
I20260926 07:22:51.092445 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.086395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.28.235.194:44013)
0926 07:22:51.087131 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:51.087147 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:51.087179 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:22:51.087294 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:51.087301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:51.087616 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:51.088194 (+ 578us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:51.088420 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.089574 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.090937 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.091789 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.091939 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.092113 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:51.092219 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 07:22:51.092318 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":277,"threads_started":1}
I20260926 07:22:51.092430 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.085987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:44013 (local address 127.0.0.1:49394)
0926 07:22:51.086212 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:51.086233 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:51.086364 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:51.087826 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:51.087830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:51.087862 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:51.088016 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:51.088023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.089726 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.089730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:51.090768 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:51.090777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.091918 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.091921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:51.092018 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.092101 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 07:22:51.092172 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:22:51.092270 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":5}
I20260926 07:22:51.092655 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.086394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:46147 (local address 127.0.0.1:55716)
0926 07:22:51.086835 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:51.086854 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:51.086976 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:51.087861 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:51.087864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:51.087882 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:51.088035 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:51.088044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.089053 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.089059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:51.090215 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:51.090225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.092009 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.092014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:51.092181 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.092216 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:22:51.092303 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:22:51.092402 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":29,"thread_start_us":217,"threads_started":1}
I20260926 07:22:51.093448 13162 tablet_bootstrap.cc:654] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:51.093518 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.087024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.28.235.195:46147)
0926 07:22:51.087353 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:51.087367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:51.087396 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:22:51.087490 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:51.087495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:51.087710 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:51.087894 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:51.088133 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.088900 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.090396 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.091833 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.092926 (+ 1093us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.093120 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:51.093179 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:22:51.093305 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":135,"threads_started":1}
I20260926 07:22:51.095045 13162 tablet_bootstrap.cc:492] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: No bootstrap required, opened a new log
I20260926 07:22:51.095119 13162 ts_tablet_manager.cc:1397] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:22:51.095537 13162 raft_consensus.cc:348] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:22:51.095640 13162 raft_consensus.cc:374] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:51.095669 13162 raft_consensus.cc:729] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ca4960845f43f8b0c66d5ca6074c84, State: Initialized, Role: FOLLOWER
I20260926 07:22:51.095835 13162 consensus_queue.cc:260] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:22:51.096146 13162 ts_tablet_manager.cc:1428] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:22:51.096218 13474 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4ae879036247dea7671b838084aeee (DEFAULT_TABLE table=test-table [id=f42aa580c12b493aaf9dca48a138c800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:51.096218 13342 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4ae879036247dea7671b838084aeee (DEFAULT_TABLE table=test-table [id=f42aa580c12b493aaf9dca48a138c800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:22:51.097610 13474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4ae879036247dea7671b838084aeee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:51.099351 13342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4ae879036247dea7671b838084aeee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:22:51.107169 13554 tablet_bootstrap.cc:492] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: Bootstrap starting.
I20260926 07:22:51.107124 13553 tablet_bootstrap.cc:492] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: Bootstrap starting.
I20260926 07:22:51.110986 13554 tablet_bootstrap.cc:654] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:51.110986 13553 tablet_bootstrap.cc:654] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: Neither blocks nor log segments found. Creating new log.
I20260926 07:22:51.112183 13553 log.cc:824] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:51.112183 13554 log.cc:824] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:22:51.115160 13553 tablet_bootstrap.cc:492] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: No bootstrap required, opened a new log
I20260926 07:22:51.115160 13554 tablet_bootstrap.cc:492] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: No bootstrap required, opened a new log
I20260926 07:22:51.115432 13554 ts_tablet_manager.cc:1397] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 07:22:51.115432 13553 ts_tablet_manager.cc:1397] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:22:51.120628 13553 raft_consensus.cc:348] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:22:51.120627 13554 raft_consensus.cc:348] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:22:51.121111 13553 raft_consensus.cc:374] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:51.121111 13554 raft_consensus.cc:374] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:22:51.121170 13553 raft_consensus.cc:729] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4784fb4e343342f5881e2135fad697ef, State: Initialized, Role: FOLLOWER
I20260926 07:22:51.121170 13554 raft_consensus.cc:729] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61de86d26384c678c85445894340a2d, State: Initialized, Role: FOLLOWER
I20260926 07:22:51.121809 13554 consensus_queue.cc:260] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:22:51.121809 13553 consensus_queue.cc:260] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:22:51.122833 13408 heartbeater.cc:503] Master 127.28.235.254:43069 was elected leader, sending a full tablet report...
I20260926 07:22:51.123039 13540 heartbeater.cc:503] Master 127.28.235.254:43069 was elected leader, sending a full tablet report...
I20260926 07:22:51.123191 13553 ts_tablet_manager.cc:1428] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: Time spent starting tablet: real 0.008s user 0.004s sys 0.005s
I20260926 07:22:51.123191 13554 ts_tablet_manager.cc:1428] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: Time spent starting tablet: real 0.008s user 0.004s sys 0.005s
I20260926 07:22:51.149294 13557 raft_consensus.cc:482] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:22:51.149623 13557 raft_consensus.cc:504] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:22:51.151190 13557 leader_election.cc:288] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a61de86d26384c678c85445894340a2d (127.28.235.194:44013), f1ca4960845f43f8b0c66d5ca6074c84 (127.28.235.193:38343)
I20260926 07:22:51.157280 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.151892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:51701 (local address 127.28.235.194:44013)
0926 07:22:51.152042 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:51.152047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:51.152264 (+ 217us) server_negotiation.cc:407] Connection header received
0926 07:22:51.152337 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:51.152342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:51.152485 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:51.152577 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:51.152976 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.153871 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.155886 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.156572 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.156720 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.156836 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:51.156993 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:22:51.157111 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:22:51.157276 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.151626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:44013 (local address 127.28.235.195:51701)
0926 07:22:51.152164 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:51.152175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:51.152255 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:51.152668 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:51.152672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:51.152688 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:51.152855 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:51.152863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.154011 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.154015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:51.155699 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:51.155714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.156716 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.156719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:51.156799 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.156825 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:22:51.156957 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 07:22:51.157073 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":188,"thread_start_us":175,"threads_started":1}
I20260926 07:22:51.157434 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.151585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38343 (local address 127.28.235.195:44523)
0926 07:22:51.151851 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:51.151869 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:22:51.152005 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:22:51.152494 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:22:51.152497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:22:51.152517 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:22:51.152762 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:51.152772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.154011 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.154015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:51.155699 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:22:51.155714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.156969 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.156972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:22:51.157120 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.157141 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:22:51.157208 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:22:51.157269 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":184}
I20260926 07:22:51.157593 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.151739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:44523 (local address 127.28.235.193:38343)
0926 07:22:51.152073 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:51.152078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:51.152093 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:22:51.152166 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:51.152168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:51.152265 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:51.152440 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:22:51.152953 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.153841 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.156025 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.156837 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.156999 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.157186 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:51.157228 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 07:22:51.157406 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":110,"threads_started":1}
I20260926 07:22:51.158248 13224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4ae879036247dea7671b838084aeee" candidate_uuid: "4784fb4e343342f5881e2135fad697ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" is_pre_election: true
I20260926 07:22:51.158511 13362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4ae879036247dea7671b838084aeee" candidate_uuid: "4784fb4e343342f5881e2135fad697ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61de86d26384c678c85445894340a2d" is_pre_election: true
I20260926 07:22:51.158671 13224 raft_consensus.cc:2454] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4784fb4e343342f5881e2135fad697ef in term 0.
I20260926 07:22:51.158906 13362 raft_consensus.cc:2454] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4784fb4e343342f5881e2135fad697ef in term 0.
I20260926 07:22:51.159196 13428 leader_election.cc:302] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4784fb4e343342f5881e2135fad697ef, f1ca4960845f43f8b0c66d5ca6074c84; no voters:
I20260926 07:22:51.159584 13557 raft_consensus.cc:2781] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:22:51.159727 13557 raft_consensus.cc:482] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:22:51.159789 13557 raft_consensus.cc:3037] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:51.161479 13557 raft_consensus.cc:504] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:22:51.161966 13557 leader_election.cc:288] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [CANDIDATE]: Term 1 election: Requested vote from peers a61de86d26384c678c85445894340a2d (127.28.235.194:44013), f1ca4960845f43f8b0c66d5ca6074c84 (127.28.235.193:38343)
I20260926 07:22:51.162448 13362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4ae879036247dea7671b838084aeee" candidate_uuid: "4784fb4e343342f5881e2135fad697ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61de86d26384c678c85445894340a2d"
I20260926 07:22:51.162448 13224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4ae879036247dea7671b838084aeee" candidate_uuid: "4784fb4e343342f5881e2135fad697ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ca4960845f43f8b0c66d5ca6074c84"
I20260926 07:22:51.162614 13362 raft_consensus.cc:3037] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:51.162614 13224 raft_consensus.cc:3037] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:22:51.164184 13362 raft_consensus.cc:2454] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4784fb4e343342f5881e2135fad697ef in term 1.
I20260926 07:22:51.164184 13224 raft_consensus.cc:2454] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4784fb4e343342f5881e2135fad697ef in term 1.
I20260926 07:22:51.164589 13428 leader_election.cc:302] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4784fb4e343342f5881e2135fad697ef, f1ca4960845f43f8b0c66d5ca6074c84; no voters:
I20260926 07:22:51.164880 13557 raft_consensus.cc:2781] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:22:51.165199 13557 raft_consensus.cc:686] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 1 LEADER]: Becoming Leader. State: Replica: 4784fb4e343342f5881e2135fad697ef, State: Running, Role: LEADER
I20260926 07:22:51.165693 13557 consensus_queue.cc:237] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:22:51.169665 13081 catalog_manager.cc:5495] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 4784fb4e343342f5881e2135fad697ef (127.28.235.195). New cstate: current_term: 1 leader_uuid: "4784fb4e343342f5881e2135fad697ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } health_report { overall_health: UNKNOWN } } }
I20260926 07:22:51.187180 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.182949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.28.235.195:46147)
0926 07:22:51.183083 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:22:51.183086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:22:51.183100 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:22:51.183227 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:22:51.183229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:22:51.183282 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:22:51.183366 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:22:51.183659 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.184278 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.185385 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:22:51.185803 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:22:51.185890 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:22:51.186174 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:22:51.186690 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:22:51.186837 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:22:51.186905 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 07:22:51.186983 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:22:51.194068 13224 raft_consensus.cc:1261] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Refusing update from remote peer 4784fb4e343342f5881e2135fad697ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:51.194072 13362 raft_consensus.cc:1261] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d [term 1 FOLLOWER]: Refusing update from remote peer 4784fb4e343342f5881e2135fad697ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:22:51.194782 13557 consensus_queue.cc:1035] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:51.194988 13561 consensus_queue.cc:1035] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:22:51.201591 13564 mvcc.cc:205] Tried to move back new op lower bound from 7333508592402591744 to 7333508592298135552. Current Snapshot: MvccSnapshot[applied={T|T < 7333508592402591744}]
I20260926 07:22:51.202178 13565 mvcc.cc:205] Tried to move back new op lower bound from 7333508592402591744 to 7333508592298135552. Current Snapshot: MvccSnapshot[applied={T|T < 7333508592402591744}]
I20260926 07:22:51.204559 13567 mvcc.cc:205] Tried to move back new op lower bound from 7333508592402591744 to 7333508592298135552. Current Snapshot: MvccSnapshot[applied={T|T < 7333508592402591744}]
I20260926 07:22:51.213891 13081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:22:51.214109 13081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:22:51.216789 13081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Sending DeleteTablet for 3 replicas of tablet 3a4ae879036247dea7671b838084aeee
I20260926 07:22:51.217936 13342 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4ae879036247dea7671b838084aeee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:51 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:49394
I20260926 07:22:51.217936 13474 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4ae879036247dea7671b838084aeee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:51 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:55716
I20260926 07:22:51.218005 13204 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4ae879036247dea7671b838084aeee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:22:51 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:59150
I20260926 07:22:51.218269 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 13120
I20260926 07:22:51.218672 13575 tablet_replica.cc:330] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: stopping tablet replica
I20260926 07:22:51.219084 13575 raft_consensus.cc:2229] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:22:51.219566 13575 raft_consensus.cc:2258] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:51.220202 13574 tablet_replica.cc:330] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: stopping tablet replica
I20260926 07:22:51.220597 13574 raft_consensus.cc:2229] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:22:51.221110 13574 raft_consensus.cc:2258] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:22:51.221664 13575 ts_tablet_manager.cc:1905] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:51.222862 13574 ts_tablet_manager.cc:1905] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:22:51.225425 13575 ts_tablet_manager.cc:1918] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:22:51.225556 13575 log.cc:1196] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/3a4ae879036247dea7671b838084aeee
I20260926 07:22:51.225927 13575 ts_tablet_manager.cc:1939] T 3a4ae879036247dea7671b838084aeee P a61de86d26384c678c85445894340a2d: Deleting consensus metadata
I20260926 07:22:51.226274 13574 ts_tablet_manager.cc:1918] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:22:51.226440 13574 log.cc:1196] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/3a4ae879036247dea7671b838084aeee
I20260926 07:22:51.226897 13574 ts_tablet_manager.cc:1939] T 3a4ae879036247dea7671b838084aeee P 4784fb4e343342f5881e2135fad697ef: Deleting consensus metadata
W20260926 07:22:51.226889 13067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:38343 (error 108)
I20260926 07:22:51.227411 13069 catalog_manager.cc:4841] TS a61de86d26384c678c85445894340a2d (127.28.235.194:44013): tablet 3a4ae879036247dea7671b838084aeee (table test-table [id=f42aa580c12b493aaf9dca48a138c800]) successfully deleted
I20260926 07:22:51.228222 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 13274
I20260926 07:22:51.228313 13066 catalog_manager.cc:4841] TS 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147): tablet 3a4ae879036247dea7671b838084aeee (table test-table [id=f42aa580c12b493aaf9dca48a138c800]) successfully deleted
I20260926 07:22:51.230907 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:22:51.230510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38343 (local address 127.0.0.1:59166)
0926 07:22:51.230713 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 07:22:51.230789 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:38343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260926 07:22:51.231096 13067 catalog_manager.cc:4584] TS f1ca4960845f43f8b0c66d5ca6074c84 (127.28.235.193:38343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a4ae879036247dea7671b838084aeee: Network error: Client connection negotiation failed: client connection to 127.28.235.193:38343: connect: Connection refused (error 111)
I20260926 07:22:51.235682 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 13411
I20260926 07:22:51.241402 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 13049
I20260926 07:23:11.248641 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43069
--webserver_interface=127.28.235.254
--webserver_port=35625
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:11.370627 13577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:11.370924 13577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:11.370965 13577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:11.375118 13577 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:23:11.375186 13577 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:23:11.375202 13577 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:23:11.375216 13577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:11.375234 13577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:11.375253 13577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:11.375269 13577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:11.375285 13577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:11.381273 13577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:43069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:11.382608 13577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:11.383805 13577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:11.394629 13582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:11.394663 13583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:11.394626 13585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:11.394753 13577 server_base.cc:1034] running on GCE node
I20260926 07:23:11.395645 13577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:11.396865 13577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:11.398203 13577 hybrid_clock.cc:648] HybridClock initialized: now 1790407391398156 us; error 69 us; skew 500 ppm
Sep 26 07:23:11 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407391, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:11.401798 13577 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:23:11.402961 13577 webserver.cc:460] Webserver started at http://127.28.235.254:35625/ using document root <none> and password file <none>
I20260926 07:23:11.403743 13577 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:11.403848 13577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:11.408716 13577 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:23:11.411623 13592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:11.413601 13577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 07:23:11.413796 13577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "2c536ffc3b9744aeafa34fd4c16c96f9"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "adb15f9f33f67541e4bfd68288823cb3"
server_key_iv: "7b6fe4491861fba90000500b84e5f062"
server_key_version: "encryptionkey@0"
I20260926 07:23:11.414462 13577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:11.428411 13577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:11.431908 13577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:11.432117 13577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:11.440201 13577 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:43069
I20260926 07:23:11.440647 13644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:43069 every 8 connection(s)
I20260926 07:23:11.441491 13577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:11.446167 13645 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:23:11.447368 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 13577
I20260926 07:23:11.447886 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:38343
--local_ip_for_outbound_sockets=127.28.235.193
--tserver_master_addrs=127.28.235.254:43069
--webserver_port=42631
--webserver_interface=127.28.235.193
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:23:11.448688 13645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Bootstrap starting.
I20260926 07:23:11.473222 13645 log.cc:824] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:11.488683 13645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:23:11.489241 13645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Bootstrap complete.
I20260926 07:23:11.495078 13645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:23:11.495609 13645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c536ffc3b9744aeafa34fd4c16c96f9, State: Initialized, Role: FOLLOWER
I20260926 07:23:11.496224 13645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:23:11.496387 13645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:11.496456 13645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:11.496562 13645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:23:11.499235 13645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:23:11.499624 13645 leader_election.cc:302] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c536ffc3b9744aeafa34fd4c16c96f9; no voters:
I20260926 07:23:11.500001 13645 leader_election.cc:288] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:23:11.500279 13649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:23:11.500679 13649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [term 3 LEADER]: Becoming Leader. State: Replica: 2c536ffc3b9744aeafa34fd4c16c96f9, State: Running, Role: LEADER
I20260926 07:23:11.501194 13649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } }
I20260926 07:23:11.501611 13645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:11.503170 13650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } } }
I20260926 07:23:11.503350 13650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:11.503153 13651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c536ffc3b9744aeafa34fd4c16c96f9. Latest consensus state: current_term: 3 leader_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c536ffc3b9744aeafa34fd4c16c96f9" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 43069 } } }
I20260926 07:23:11.503505 13651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:11.503984 13662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:11.509963 13662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:11.510783 13662 catalog_manager.cc:1259] Loaded cluster ID: ce42ce341f98431bbb26f34b4cac36d6
I20260926 07:23:11.510874 13662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:11.514060 13662 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:11.514816 13662 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Loaded TSK: 0
I20260926 07:23:11.514925 13662 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c536ffc3b9744aeafa34fd4c16c96f9: Loaded TSK: 1
F20260926 07:23:11.515671 13662 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f286162d9ad google::LogMessage::Fail() at ??:0
@ 0x7f286163088a google::LogMessage::SendToLog() at ??:0
@ 0x7f286162d4de google::LogMessage::Flush() at ??:0
@ 0x7f286162ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f28642f118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f28642e9f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f286bf927bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f286bf6c99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f286bf6d178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f286bfaedf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f286bfcd82b std::function<>::operator()() at ??:0
@ 0x7f286bf6cb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f286bf6df3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f286bf6a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f286bfae640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f286387f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f286387fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f28638813f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2863870512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2864eb46db start_thread at ??:0
@ 0x7f286012b61f clone at ??:0
W20260926 07:23:11.616499 13647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:11.616801 13647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:11.616850 13647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:11.621390 13647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:11.621490 13647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:11.621618 13647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:23:11.627617 13647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:38343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=42631
--tserver_master_addrs=127.28.235.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:11.629129 13647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:11.630400 13647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:11.642007 13692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:11.642185 13693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:11.641999 13695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:11.642992 13647 server_base.cc:1034] running on GCE node
I20260926 07:23:11.643554 13647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:11.644459 13647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:11.645759 13647 hybrid_clock.cc:648] HybridClock initialized: now 1790407391645714 us; error 71 us; skew 500 ppm
Sep 26 07:23:11 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407391, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:11.648998 13647 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:23:11.650197 13647 webserver.cc:460] Webserver started at http://127.28.235.193:42631/ using document root <none> and password file <none>
I20260926 07:23:11.650907 13647 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:11.650974 13647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:11.655967 13647 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:23:11.659128 13702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:11.661092 13647 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 07:23:11.661289 13647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "f1ca4960845f43f8b0c66d5ca6074c84"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "fe500d24e310c732ed99a38ce8f49a91"
server_key_iv: "fc876c52eaae92c3c731c57d5c4ef456"
server_key_version: "encryptionkey@0"
I20260926 07:23:11.661954 13647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:11.679522 13647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:11.683228 13647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:11.683450 13647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:11.684223 13647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:11.685485 13709 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:23:11.691289 13647 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:23:11.691388 13647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 07:23:11.691493 13647 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:23:11.693387 13647 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:23:11.693455 13647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260926 07:23:11.693786 13709 tablet_bootstrap.cc:492] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: Bootstrap starting.
I20260926 07:23:11.705519 13647 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:38343
I20260926 07:23:11.705799 13816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:38343 every 8 connection(s)
I20260926 07:23:11.706712 13647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:23:11.707739 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 13647
I20260926 07:23:11.708212 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:44013
--local_ip_for_outbound_sockets=127.28.235.194
--tserver_master_addrs=127.28.235.254:43069
--webserver_port=45703
--webserver_interface=127.28.235.194
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:23:11.716344 13709 log.cc:824] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:11.726353 13709 tablet_bootstrap.cc:492] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:23:11.726851 13709 tablet_bootstrap.cc:492] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: Bootstrap complete.
I20260926 07:23:11.728360 13709 ts_tablet_manager.cc:1397] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.005s
I20260926 07:23:11.731914 13709 raft_consensus.cc:348] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:23:11.732522 13709 raft_consensus.cc:729] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ca4960845f43f8b0c66d5ca6074c84, State: Initialized, Role: FOLLOWER
I20260926 07:23:11.733099 13709 consensus_queue.cc:260] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:23:11.734206 13709 ts_tablet_manager.cc:1428] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260926 07:23:11.766952 13820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:23:11.710306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:42871)
0926 07:23:11.711912 (+ 1606us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:11.711975 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:11.712830 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:11.765962 (+ 53132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":166,"threads_started":1}
W20260926 07:23:11.767627 13817 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:43069 (0 consecutive failures): Network error: Failed to ping master at 127.28.235.254:43069: Client connection negotiation failed: client connection to 127.28.235.254:43069: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:23:11.768407 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:11.768132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:55447)
0926 07:23:11.768268 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:11.768341 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 07:23:11.769169 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:11.768895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:41729)
0926 07:23:11.769035 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:11.769107 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 07:23:11.769470 13817 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:23:11.833189 13819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:11.833485 13819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:11.833539 13819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:11.837749 13819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:11.837814 13819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:11.837877 13819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:23:11.843633 13819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:44013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=45703
--tserver_master_addrs=127.28.235.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:11.844817 13819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:11.846145 13819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:11.859292 13828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:11.859324 13827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:11.859644 13819 server_base.cc:1034] running on GCE node
W20260926 07:23:11.859301 13830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:11.860280 13819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:11.861001 13819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:11.862236 13819 hybrid_clock.cc:648] HybridClock initialized: now 1790407391862212 us; error 50 us; skew 500 ppm
Sep 26 07:23:11 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407391, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:11.865391 13819 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:23:11.866515 13819 webserver.cc:460] Webserver started at http://127.28.235.194:45703/ using document root <none> and password file <none>
I20260926 07:23:11.867126 13819 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:11.867188 13819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:11.871682 13819 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:23:11.874505 13837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:11.877445 13819 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:23:11.877703 13819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "a61de86d26384c678c85445894340a2d"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "7688e30ef3a20d7a355ee8b3d7ce08fa"
server_key_iv: "5617fe176e035f1b7f64a8be58f4e9d5"
server_key_version: "encryptionkey@0"
I20260926 07:23:11.878394 13819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:11.903481 13819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:11.907414 13819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:11.907660 13819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:11.908353 13819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:11.909354 13819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:11.909426 13819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:11.909494 13819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:11.909543 13819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:11.921089 13819 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:44013
I20260926 07:23:11.921190 13950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:44013 every 8 connection(s)
I20260926 07:23:11.922199 13819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:23:11.927184 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:11.924773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:56187)
0926 07:23:11.925939 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:11.926107 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":885,"thread_start_us":675,"threads_started":1}
W20260926 07:23:11.927732 13951 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:43069 (0 consecutive failures): Network error: Failed to ping master at 127.28.235.254:43069: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
I20260926 07:23:11.928663 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:11.928304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:43801)
0926 07:23:11.928485 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:11.928562 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 07:23:11.929132 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 13819
I20260926 07:23:11.929656 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:46147
--local_ip_for_outbound_sockets=127.28.235.195
--tserver_master_addrs=127.28.235.254:43069
--webserver_port=42849
--webserver_interface=127.28.235.195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:23:11.929692 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:11.929396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:47729)
0926 07:23:11.929542 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:11.929610 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 07:23:11.930141 13951 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:23:12.087597 13954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:12.087893 13954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:12.087939 13954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:12.092396 13954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:12.092479 13954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:12.092571 13954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:23:12.098476 13954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:46147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=42849
--tserver_master_addrs=127.28.235.254:43069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:12.099625 13954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:12.100673 13954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:12.111620 13960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:12.111616 13959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:12.111613 13962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:12.111732 13954 server_base.cc:1034] running on GCE node
I20260926 07:23:12.112382 13954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:12.113145 13954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:12.114353 13954 hybrid_clock.cc:648] HybridClock initialized: now 1790407392114340 us; error 36 us; skew 500 ppm
Sep 26 07:23:12 dist-test-slave-7zlw krb5kdc[12501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407392, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:12.117326 13954 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:23:12.118372 13954 webserver.cc:460] Webserver started at http://127.28.235.195:42849/ using document root <none> and password file <none>
I20260926 07:23:12.119051 13954 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:12.119146 13954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:12.123209 13954 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:23:12.125478 13969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:12.126950 13954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:12.127195 13954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "4784fb4e343342f5881e2135fad697ef"
format_stamp: "Formatted at 2026-09-26 07:22:29 on dist-test-slave-7zlw"
server_key: "56746f5066244ac8ccca976223619b69"
server_key_iv: "d8cf590b4d6f844b73a568e0bca41e72"
server_key_version: "encryptionkey@0"
I20260926 07:23:12.127779 13954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:12.157366 13954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:12.161006 13954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:12.161262 13954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:12.161916 13954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:12.162776 13954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:12.162858 13954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:12.162933 13954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:12.162986 13954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:12.172911 13954 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:46147
I20260926 07:23:12.172932 14082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:46147 every 8 connection(s)
I20260926 07:23:12.174069 13954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:23:12.178198 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:12.176526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:58481)
0926 07:23:12.177356 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:12.177470 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":119,"threads_started":1}
W20260926 07:23:12.178791 14083 heartbeater.cc:644] Failed to heartbeat to 127.28.235.254:43069 (0 consecutive failures): Network error: Failed to ping master at 127.28.235.254:43069: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
I20260926 07:23:12.179452 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:12.179209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:48591)
0926 07:23:12.179335 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:12.179393 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 07:23:12.180060 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 13954
I20260926 07:23:12.180174 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:12.179896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:55731)
0926 07:23:12.180045 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:12.180104 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 07:23:12.180335 14083 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:23:12.770653 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:12.770181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:37027)
0926 07:23:12.770478 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:12.770554 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260926 07:23:12.931468 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:12.930955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:45019)
0926 07:23:12.931277 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:12.931343 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":76,"threads_started":1}
I20260926 07:23:13.081471 14089 raft_consensus.cc:482] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:13.081784 14089 raft_consensus.cc:504] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:23:13.083065 14089 leader_election.cc:288] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a61de86d26384c678c85445894340a2d (127.28.235.194:44013), 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147)
Sep 26 07:23:13 dist-test-slave-7zlw krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407391, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.195@KRBTEST.COM
Sep 26 07:23:13 dist-test-slave-7zlw krb5kdc[12501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407391, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.194@KRBTEST.COM
I20260926 07:23:13.109457 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:13.083565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:55583 (local address 127.28.235.195:46147)
0926 07:23:13.083892 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:13.083908 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:13.084097 (+ 189us) server_negotiation.cc:407] Connection header received
0926 07:23:13.084806 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:13.084812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:13.085213 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:13.085528 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:13.086899 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:13.087929 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:13.089296 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:13.089769 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:13.089955 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:13.103361 (+ 13406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:13.103371 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:13.103390 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:13.103435 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:13.105896 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:13.106757 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:13.106764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:13.106770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:13.106925 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:13.107371 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:13.107374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:13.107376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:13.107955 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:13.108102 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:13.108912 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 07:23:13.109269 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
I20260926 07:23:13.109457 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:13.083577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:48547 (local address 127.28.235.194:44013)
0926 07:23:13.084024 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:13.084043 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:13.084079 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:23:13.084874 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:13.084880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:13.085278 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:13.085581 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:13.086900 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:13.087958 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:13.089308 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:13.089805 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:13.090000 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:13.103397 (+ 13397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:13.103403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:13.103414 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:13.103450 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:13.105896 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:13.106757 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:13.106764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:13.106770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:13.106925 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:13.107371 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:13.107374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:13.107376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:13.107955 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:13.108102 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:13.108912 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 07:23:13.109269 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":104,"threads_started":1}
I20260926 07:23:13.109436 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:13.083425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:44013 (local address 127.28.235.193:48547)
0926 07:23:13.083675 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:13.083695 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:13.083811 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:13.085610 (+ 1799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:13.085616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:13.086206 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:13.086715 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:13.086731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:13.088094 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:13.088102 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:13.089169 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:13.089180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:13.089940 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:13.089944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:13.090162 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:13.101073 (+ 10911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:13.101099 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:13.103136 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:13.106057 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:13.106065 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:13.106081 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:13.106626 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:13.107034 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:13.107036 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:13.107051 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:13.107264 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:13.108112 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:13.108120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:13.108529 (+ 409us) client_negotiation.cc:769] Sending connection context
0926 07:23:13.108767 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 07:23:13.109218 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":249,"spinlock_wait_cycles":49664}
I20260926 07:23:13.109436 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:13.083425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:46147 (local address 127.28.235.193:55583)
0926 07:23:13.083965 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:13.083986 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:13.084118 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:13.085608 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:13.085614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:13.086205 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:13.086715 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:13.086731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:13.088095 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:13.088098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:13.089168 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:13.089179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:13.089901 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:13.089905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:13.090154 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:13.100987 (+ 10833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:13.101062 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:13.102974 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:13.106057 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:13.106065 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:13.106081 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:13.106626 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:13.107067 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:13.107069 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:13.107071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:13.107264 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:13.108112 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:13.108120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:13.108529 (+ 409us) client_negotiation.cc:769] Sending connection context
0926 07:23:13.108767 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 07:23:13.109270 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":401,"spinlock_wait_cycles":15616,"thread_start_us":83,"threads_started":1}
W20260926 07:23:13.112190 13703 leader_election.cc:341] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147): Not found: Tablet not found: 3a4ae879036247dea7671b838084aeee
W20260926 07:23:13.112768 13706 leader_election.cc:341] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a61de86d26384c678c85445894340a2d (127.28.235.194:44013): Not found: Tablet not found: 3a4ae879036247dea7671b838084aeee
I20260926 07:23:13.112838 13706 leader_election.cc:302] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1ca4960845f43f8b0c66d5ca6074c84; no voters: 4784fb4e343342f5881e2135fad697ef, a61de86d26384c678c85445894340a2d
I20260926 07:23:13.113152 14089 raft_consensus.cc:2726] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:23:13.181548 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:13.181097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:39769)
0926 07:23:13.181379 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:13.181440 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260926 07:23:13.771991 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:13.771532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:52263)
0926 07:23:13.771824 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:13.771891 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260926 07:23:13.932881 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:13.932404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:40053)
0926 07:23:13.932703 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:13.932765 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":145,"threads_started":1}
I20260926 07:23:14.182852 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:14.182419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:50285)
0926 07:23:14.182684 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:14.182739 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260926 07:23:14.773325 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:14.772835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:56307)
0926 07:23:14.773168 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:14.773228 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":76,"threads_started":1}
I20260926 07:23:14.934183 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:14.933728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:41909)
0926 07:23:14.933972 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:14.934063 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":107,"threads_started":1}
I20260926 07:23:15.030967 14099 raft_consensus.cc:482] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:15.031113 14099 raft_consensus.cc:504] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:23:15.031613 14099 leader_election.cc:288] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a61de86d26384c678c85445894340a2d (127.28.235.194:44013), 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147)
W20260926 07:23:15.032483 13703 leader_election.cc:341] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147): Not found: Tablet not found: 3a4ae879036247dea7671b838084aeee
W20260926 07:23:15.032567 13706 leader_election.cc:341] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a61de86d26384c678c85445894340a2d (127.28.235.194:44013): Not found: Tablet not found: 3a4ae879036247dea7671b838084aeee
I20260926 07:23:15.032612 13703 leader_election.cc:302] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1ca4960845f43f8b0c66d5ca6074c84; no voters: 4784fb4e343342f5881e2135fad697ef, a61de86d26384c678c85445894340a2d
I20260926 07:23:15.032822 14099 raft_consensus.cc:2726] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:23:15.184149 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:15.183692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:45565)
0926 07:23:15.183984 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:15.184054 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
I20260926 07:23:15.774711 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:15.774244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:54075)
0926 07:23:15.774542 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:15.774608 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260926 07:23:15.935384 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:15.934950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:33137)
0926 07:23:15.935209 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:15.935287 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260926 07:23:16.185523 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:16.184998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:34843)
0926 07:23:16.185314 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:16.185415 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":74,"threads_started":1}
I20260926 07:23:16.776051 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:16.775611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:58551)
0926 07:23:16.775884 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:16.775939 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":71,"threads_started":1}
I20260926 07:23:16.936760 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:16.936261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:55663)
0926 07:23:16.936566 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:16.936648 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
I20260926 07:23:17.186820 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:17.186413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:45563)
0926 07:23:17.186672 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:17.186727 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
I20260926 07:23:17.420703 14107 raft_consensus.cc:482] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:17.420835 14107 raft_consensus.cc:504] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:23:17.421339 14107 leader_election.cc:288] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a61de86d26384c678c85445894340a2d (127.28.235.194:44013), 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147)
W20260926 07:23:17.422145 13706 leader_election.cc:341] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a61de86d26384c678c85445894340a2d (127.28.235.194:44013): Not found: Tablet not found: 3a4ae879036247dea7671b838084aeee
W20260926 07:23:17.422227 13703 leader_election.cc:341] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147): Not found: Tablet not found: 3a4ae879036247dea7671b838084aeee
I20260926 07:23:17.422359 13703 leader_election.cc:302] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1ca4960845f43f8b0c66d5ca6074c84; no voters: 4784fb4e343342f5881e2135fad697ef, a61de86d26384c678c85445894340a2d
I20260926 07:23:17.422502 14107 raft_consensus.cc:2726] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:23:17.777406 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:17.776913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:52687)
0926 07:23:17.777221 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:17.777289 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":148,"threads_started":1}
I20260926 07:23:17.938069 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:17.937677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:39817)
0926 07:23:17.937917 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:17.937971 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260926 07:23:18.188086 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:18.187660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:59231)
0926 07:23:18.187932 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:18.187994 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":95,"threads_started":1}
I20260926 07:23:18.778957 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:18.778415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:40705)
0926 07:23:18.778730 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:18.778812 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":122,"threads_started":1}
I20260926 07:23:18.939461 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:18.938946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:48803)
0926 07:23:18.939257 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:18.939318 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
I20260926 07:23:19.189653 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:19.188984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:35587)
0926 07:23:19.189486 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:19.189554 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":87,"threads_started":1}
I20260926 07:23:19.780377 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:19.779887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:34463)
0926 07:23:19.780169 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:19.780230 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260926 07:23:19.941381 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:19.940646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:53713)
0926 07:23:19.941085 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:19.941241 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":96,"threads_started":1}
I20260926 07:23:20.191614 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:20.190918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:52711)
0926 07:23:20.191350 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:20.191493 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":104,"threads_started":1}
I20260926 07:23:20.782269 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:20.781654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:50677)
0926 07:23:20.782061 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:20.782131 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":119,"threads_started":1}
I20260926 07:23:20.942850 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:20.942383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:46733)
0926 07:23:20.942681 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:20.942747 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":111,"threads_started":1}
I20260926 07:23:21.193243 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:21.192703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:44239)
0926 07:23:21.193067 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:21.193146 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
I20260926 07:23:21.783651 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:21.783204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:42943)
0926 07:23:21.783497 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:21.783554 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":77,"threads_started":1}
I20260926 07:23:21.944537 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:21.943850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:59511)
0926 07:23:21.944238 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:21.944379 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":101,"threads_started":1}
I20260926 07:23:22.195019 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:22.194373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:48233)
0926 07:23:22.194786 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:22.194880 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":129,"threads_started":1}
I20260926 07:23:22.476002 14123 raft_consensus.cc:482] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:22.476227 14123 raft_consensus.cc:504] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a61de86d26384c678c85445894340a2d" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 44013 } } peers { permanent_uuid: "4784fb4e343342f5881e2135fad697ef" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 46147 } } peers { permanent_uuid: "f1ca4960845f43f8b0c66d5ca6074c84" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38343 } }
I20260926 07:23:22.476778 14123 leader_election.cc:288] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a61de86d26384c678c85445894340a2d (127.28.235.194:44013), 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147)
W20260926 07:23:22.477725 13703 leader_election.cc:341] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4784fb4e343342f5881e2135fad697ef (127.28.235.195:46147): Not found: Tablet not found: 3a4ae879036247dea7671b838084aeee
W20260926 07:23:22.477905 13706 leader_election.cc:341] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a61de86d26384c678c85445894340a2d (127.28.235.194:44013): Not found: Tablet not found: 3a4ae879036247dea7671b838084aeee
I20260926 07:23:22.477998 13706 leader_election.cc:302] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1ca4960845f43f8b0c66d5ca6074c84; no voters: 4784fb4e343342f5881e2135fad697ef, a61de86d26384c678c85445894340a2d
I20260926 07:23:22.478216 14123 raft_consensus.cc:2726] T 3a4ae879036247dea7671b838084aeee P f1ca4960845f43f8b0c66d5ca6074c84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:23:22.785319 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:22.784651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:50943)
0926 07:23:22.785007 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:22.785169 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
I20260926 07:23:22.945926 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:22.945465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:46561)
0926 07:23:22.945763 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:22.945824 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
I20260926 07:23:23.196480 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:23.195971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:33607)
0926 07:23:23.196266 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:23.196364 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":95,"threads_started":1}
I20260926 07:23:23.786954 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:23.786309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:55613)
0926 07:23:23.786641 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:23.786703 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":81,"threads_started":1}
I20260926 07:23:23.947471 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:23.946968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:34511)
0926 07:23:23.947259 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:23.947322 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260926 07:23:24.197943 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:24.197392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:48265)
0926 07:23:24.197676 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:24.197741 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260926 07:23:24.788321 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:24.787902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:33575)
0926 07:23:24.788164 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:24.788224 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260926 07:23:24.948760 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:24.948324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:40471)
0926 07:23:24.948612 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:24.948668 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
I20260926 07:23:25.199568 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:25.198790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:50653)
0926 07:23:25.199335 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:25.199431 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":171,"threads_started":1}
I20260926 07:23:25.790081 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:25.789408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:60797)
0926 07:23:25.789798 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:25.789886 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":96,"threads_started":1}
I20260926 07:23:25.950665 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:25.950003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:58859)
0926 07:23:25.950444 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:25.950543 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":107,"threads_started":1}
I20260926 07:23:26.200932 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:26.200446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.195:41345)
0926 07:23:26.200743 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:26.200822 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
I20260926 07:23:26.791651 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:26.791216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.193:50615)
0926 07:23:26.791496 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:26.791553 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
I20260926 07:23:26.952399 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:26.951950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:43069 (local address 127.28.235.194:36033)
0926 07:23:26.952226 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:26.952298 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.254:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260926 07:23:27.186903 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 13647
I20260926 07:23:27.193915 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 13819
I20260926 07:23:27.199736 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 13954
2026-09-26T07:23:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59469 ms)
[----------] 30 tests from SecurityITest (253008 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:27 dist-test-slave-7zlw krb5kdc[14140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:27 dist-test-slave-7zlw krb5kdc[14140](info): set up 2 sockets
Sep 26 07:23:27 dist-test-slave-7zlw krb5kdc[14140](info): commencing operation
krb5kdc: starting...
W20260926 07:23:28.276965 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:28 dist-test-slave-7zlw krb5kdc[14140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:23:28.412098 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:38195
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:38195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:28.524699 14153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:28.525048 14153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:28.525152 14153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:28.529381 14153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:28.529467 14153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:28.529505 14153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:28.529565 14153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:28.529603 14153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:28.535140 14153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:38195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:38195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:28.536284 14153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:28.537413 14153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:28.546743 14158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:28.546810 14159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:28.546852 14161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:28.547043 14153 server_base.cc:1034] running on GCE node
I20260926 07:23:28.547680 14153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:28.548851 14153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:28.550081 14153 hybrid_clock.cc:648] HybridClock initialized: now 1790407408550066 us; error 31 us; skew 500 ppm
Sep 26 07:23:28 dist-test-slave-7zlw krb5kdc[14140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407408, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:28.552977 14153 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:23:28.554028 14153 webserver.cc:460] Webserver started at http://127.28.235.254:36325/ using document root <none> and password file <none>
I20260926 07:23:28.554591 14153 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:28.554651 14153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:28.554828 14153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:28.556906 14153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "5990da145aee45d1b1085fefd8850ba1"
format_stamp: "Formatted at 2026-09-26 07:23:28 on dist-test-slave-7zlw"
server_key: "e91e4fabdad631095cea59f02957472e"
server_key_iv: "cc5dd5d9b35e0529a6a6ffb0282a72d3"
server_key_version: "encryptionkey@0"
I20260926 07:23:28.557545 14153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "5990da145aee45d1b1085fefd8850ba1"
format_stamp: "Formatted at 2026-09-26 07:23:28 on dist-test-slave-7zlw"
server_key: "e91e4fabdad631095cea59f02957472e"
server_key_iv: "cc5dd5d9b35e0529a6a6ffb0282a72d3"
server_key_version: "encryptionkey@0"
I20260926 07:23:28.561533 14153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:23:28.564105 14168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:28.565465 14153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:28.565603 14153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "5990da145aee45d1b1085fefd8850ba1"
format_stamp: "Formatted at 2026-09-26 07:23:28 on dist-test-slave-7zlw"
server_key: "e91e4fabdad631095cea59f02957472e"
server_key_iv: "cc5dd5d9b35e0529a6a6ffb0282a72d3"
server_key_version: "encryptionkey@0"
I20260926 07:23:28.565726 14153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:28.604694 14153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:28.608301 14153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:28.608492 14153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:28.615978 14153 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:38195
I20260926 07:23:28.616003 14220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:38195 every 8 connection(s)
I20260926 07:23:28.616997 14153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:28.619647 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 14153
I20260926 07:23:28.619767 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:28.620091 29615 external_mini_cluster.cc:1428] Setting key c3346581f0fc1b2376c073da037d6d04
I20260926 07:23:28.620167 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:28.627197 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1: Bootstrap starting.
Sep 26 07:23:28 dist-test-slave-7zlw krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:28.629786 14221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:28.630609 14221 log.cc:824] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:28.633158 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1: No bootstrap required, opened a new log
I20260926 07:23:28.635509 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:28.621609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.28.235.254:38195)
0926 07:23:28.622092 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:28.622112 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:28.622151 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:23:28.623099 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:28.623120 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:28.623503 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:28.623858 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:28.624654 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:28.625783 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:28.626685 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:28.627165 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:28.627337 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:28.629987 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:28.629992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:28.630007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:28.630036 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:28.632160 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:28.632610 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:28.632617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:28.632623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:28.632784 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:28.633060 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:28.633063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:28.633065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:28.633659 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:28.633886 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:28.634147 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 07:23:28.634469 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":171,"threads_started":1}
I20260926 07:23:28.638168 14221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5990da145aee45d1b1085fefd8850ba1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38195 } }
I20260926 07:23:28.638405 14221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:28.638442 14221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5990da145aee45d1b1085fefd8850ba1, State: Initialized, Role: FOLLOWER
I20260926 07:23:28.638813 14221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5990da145aee45d1b1085fefd8850ba1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38195 } }
I20260926 07:23:28.638931 14221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:28.638972 14221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:28.639041 14221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:28.639981 14221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5990da145aee45d1b1085fefd8850ba1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38195 } }
I20260926 07:23:28.640295 14221 leader_election.cc:302] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5990da145aee45d1b1085fefd8850ba1; no voters:
I20260926 07:23:28.640568 14221 leader_election.cc:288] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:28.640784 14226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:28.641057 14226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [term 1 LEADER]: Becoming Leader. State: Replica: 5990da145aee45d1b1085fefd8850ba1, State: Running, Role: LEADER
I20260926 07:23:28.641417 14226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5990da145aee45d1b1085fefd8850ba1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38195 } }
I20260926 07:23:28.641947 14221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:28.643014 14227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5990da145aee45d1b1085fefd8850ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5990da145aee45d1b1085fefd8850ba1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38195 } } }
I20260926 07:23:28.643146 14227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:28.643481 14228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5990da145aee45d1b1085fefd8850ba1. Latest consensus state: current_term: 1 leader_uuid: "5990da145aee45d1b1085fefd8850ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5990da145aee45d1b1085fefd8850ba1" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38195 } } }
I20260926 07:23:28.643573 14228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:28.643641 14235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:28.646482 14235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:28.652333 14235 catalog_manager.cc:1347] Generated new cluster ID: 0b89d1490715400a836ed3494d4eed77
I20260926 07:23:28.652422 14235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:28.678495 14235 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:28.679492 14235 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:28.686802 14235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1: Generated new TSK 0
I20260926 07:23:28.687538 14235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:23:28.721546 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:28.841820 14247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:28.842110 14247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:28.842157 14247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:28.846530 14247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:28.846602 14247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:28.846698 14247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:23:28.852262 14247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:28.853467 14247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:28.854545 14247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:28.864526 14255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:28.864494 14253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:28.864494 14252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:28.864920 14247 server_base.cc:1034] running on GCE node
I20260926 07:23:28.865341 14247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:28.866035 14247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:28.867223 14247 hybrid_clock.cc:648] HybridClock initialized: now 1790407408867190 us; error 47 us; skew 500 ppm
Sep 26 07:23:28 dist-test-slave-7zlw krb5kdc[14140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407408, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:28.870015 14247 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:23:28.870924 14247 webserver.cc:460] Webserver started at http://127.28.235.193:34209/ using document root <none> and password file <none>
I20260926 07:23:28.871490 14247 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:28.871577 14247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:28.871809 14247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:28.873963 14247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "0b3d12a6aee74be7a221080d991ae8bd"
format_stamp: "Formatted at 2026-09-26 07:23:28 on dist-test-slave-7zlw"
server_key: "72f470d6dc36a29bfb822ca10ee9ac5b"
server_key_iv: "3b6de9e29b8c3ab79c33d65cc8732f05"
server_key_version: "encryptionkey@0"
I20260926 07:23:28.874536 14247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "0b3d12a6aee74be7a221080d991ae8bd"
format_stamp: "Formatted at 2026-09-26 07:23:28 on dist-test-slave-7zlw"
server_key: "72f470d6dc36a29bfb822ca10ee9ac5b"
server_key_iv: "3b6de9e29b8c3ab79c33d65cc8732f05"
server_key_version: "encryptionkey@0"
I20260926 07:23:28.878356 14247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:23:28.880820 14262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:28.882175 14247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:23:28.882373 14247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "0b3d12a6aee74be7a221080d991ae8bd"
format_stamp: "Formatted at 2026-09-26 07:23:28 on dist-test-slave-7zlw"
server_key: "72f470d6dc36a29bfb822ca10ee9ac5b"
server_key_iv: "3b6de9e29b8c3ab79c33d65cc8732f05"
server_key_version: "encryptionkey@0"
I20260926 07:23:28.882499 14247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:28.928658 14247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:28.932380 14247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:28.932615 14247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:28.933328 14247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:28.934250 14247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:28.934332 14247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:28.934405 14247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:28.934458 14247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:28.944877 14247 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:46617
I20260926 07:23:28.944897 14375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:46617 every 8 connection(s)
I20260926 07:23:28.945892 14247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:23:28.949695 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 14247
I20260926 07:23:28.949838 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:23:28.950146 29615 external_mini_cluster.cc:1428] Setting key 58de5afcf61c88b1d1a8068b24c38671
Sep 26 07:23:28 dist-test-slave-7zlw krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407408, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:23:28.962039 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:28.947904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:57923 (local address 127.28.235.254:38195)
0926 07:23:28.948077 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:28.948081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:28.948858 (+ 777us) server_negotiation.cc:407] Connection header received
0926 07:23:28.950697 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:28.950700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:28.950758 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:28.950868 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:28.952226 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:28.952808 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:28.954167 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:28.954606 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:28.954728 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:28.957599 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:28.957605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:28.957608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:28.957641 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:28.959441 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:28.960004 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:28.960011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:28.960015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:28.960117 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:28.960593 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:28.960597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:28.960599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:28.960809 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:28.960920 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:28.961718 (+ 798us) server_negotiation.cc:299] Negotiation successful
0926 07:23:28.961861 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 07:23:28.962960 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:28.948271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:38195 (local address 127.28.235.193:57923)
0926 07:23:28.948711 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:28.948753 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:28.950433 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:28.951045 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:28.951052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:28.951394 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:28.952038 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:28.952060 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:28.952953 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:28.952959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:28.954032 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:28.954042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:28.954756 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:28.954759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:28.954904 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:28.955604 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:28.955627 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:28.957408 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:28.959586 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:28.959591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:28.959606 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:28.959874 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:28.960245 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:28.960251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:28.960254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:28.960464 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:28.960930 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:28.960961 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:28.961392 (+ 431us) client_negotiation.cc:769] Sending connection context
0926 07:23:28.961653 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 07:23:28.962027 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:23:28.964480 14376 heartbeater.cc:347] Connected to a master server at 127.28.235.254:38195
I20260926 07:23:28.964756 14376 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:28.965399 14376 heartbeater.cc:511] Master 127.28.235.254:38195 requested a full tablet report, sending...
I20260926 07:23:28.967139 14181 ts_manager.cc:194] Registered new tserver with Master: 0b3d12a6aee74be7a221080d991ae8bd (127.28.235.193:46617)
I20260926 07:23:28.968981 14181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:57923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:23:28.980758 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:29.107695 14381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:29.107959 14381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:29.108002 14381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:29.112145 14381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:29.112205 14381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:29.112309 14381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:23:29.118803 14381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:29.119961 14381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:29.121100 14381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:29.131136 14386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:29.131188 14389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:29.131136 14387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:29.131209 14381 server_base.cc:1034] running on GCE node
I20260926 07:23:29.131879 14381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:29.132543 14381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:29.133778 14381 hybrid_clock.cc:648] HybridClock initialized: now 1790407409133755 us; error 27 us; skew 500 ppm
Sep 26 07:23:29 dist-test-slave-7zlw krb5kdc[14140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407409, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:29.136759 14381 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:23:29.137669 14381 webserver.cc:460] Webserver started at http://127.28.235.194:34855/ using document root <none> and password file <none>
I20260926 07:23:29.138214 14381 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:29.138274 14381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:29.138441 14381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:29.140419 14381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f"
format_stamp: "Formatted at 2026-09-26 07:23:29 on dist-test-slave-7zlw"
server_key: "85c74076c3876676344dadebc20f27f7"
server_key_iv: "52fc696c2d740c9620be7c4ca5785aee"
server_key_version: "encryptionkey@0"
I20260926 07:23:29.140928 14381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f"
format_stamp: "Formatted at 2026-09-26 07:23:29 on dist-test-slave-7zlw"
server_key: "85c74076c3876676344dadebc20f27f7"
server_key_iv: "52fc696c2d740c9620be7c4ca5785aee"
server_key_version: "encryptionkey@0"
I20260926 07:23:29.145112 14381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:23:29.147631 14396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:29.148994 14381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:23:29.149199 14381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f"
format_stamp: "Formatted at 2026-09-26 07:23:29 on dist-test-slave-7zlw"
server_key: "85c74076c3876676344dadebc20f27f7"
server_key_iv: "52fc696c2d740c9620be7c4ca5785aee"
server_key_version: "encryptionkey@0"
I20260926 07:23:29.149312 14381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:29.163335 14381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:29.166467 14381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:29.166682 14381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:29.167351 14381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:29.168193 14381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:29.168241 14381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:29.168315 14381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:29.168354 14381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:29.177977 14381 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:35597
I20260926 07:23:29.177995 14509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:35597 every 8 connection(s)
I20260926 07:23:29.178915 14381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:23:29.187721 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 14381
I20260926 07:23:29.187849 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:23:29.188158 29615 external_mini_cluster.cc:1428] Setting key afed6a5ce9ad4c5c1e6787c1e8250ddd
Sep 26 07:23:29 dist-test-slave-7zlw krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407409, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:29.193383 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.180828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:38813 (local address 127.28.235.254:38195)
0926 07:23:29.180987 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:29.180996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:29.181816 (+ 820us) server_negotiation.cc:407] Connection header received
0926 07:23:29.183174 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:29.183178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:29.183236 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:29.183326 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:29.184470 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.185007 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.186306 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.186548 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.186666 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.189551 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:29.189554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:29.189556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:29.189580 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:29.191178 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:29.191747 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:29.191755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:29.191757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:29.191832 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:29.192197 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:29.192200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:29.192202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:29.192399 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:29.192517 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:29.193078 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 07:23:29.193212 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:23:29.194286 14512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.181206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:38195 (local address 127.28.235.194:38813)
0926 07:23:29.181647 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:29.181689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:29.182957 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:29.183488 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:29.183494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:29.183849 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:29.184319 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.184331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.185175 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.185183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.186169 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.186179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.186706 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.186711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.186953 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.187580 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:29.187604 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:29.189367 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:29.191320 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:29.191327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:29.191345 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:29.191629 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:29.191925 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:29.191929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:29.191931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:29.192089 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:29.192517 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:29.192525 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:29.192794 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 07:23:29.193039 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 07:23:29.193418 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260926 07:23:29.195502 14510 heartbeater.cc:347] Connected to a master server at 127.28.235.254:38195
I20260926 07:23:29.195760 14510 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:29.196260 14510 heartbeater.cc:511] Master 127.28.235.254:38195 requested a full tablet report, sending...
I20260926 07:23:29.197444 14181 ts_manager.cc:194] Registered new tserver with Master: 9f0725e3ec3d4d9ebdd34f3b1cfe158f (127.28.235.194:35597)
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:23:29.198202 14181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:38813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:23:29.220839 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38195
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:29.340481 14515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:29.340797 14515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:29.340847 14515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:29.345992 14515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:29.346072 14515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:29.346171 14515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:23:29.352244 14515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:29.353745 14515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:29.354861 14515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:29.365191 14521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:29.365170 14520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:29.365170 14523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:29.365502 14515 server_base.cc:1034] running on GCE node
I20260926 07:23:29.365963 14515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:29.366686 14515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:29.367910 14515 hybrid_clock.cc:648] HybridClock initialized: now 1790407409367882 us; error 41 us; skew 500 ppm
Sep 26 07:23:29 dist-test-slave-7zlw krb5kdc[14140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407409, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:29.370926 14515 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:23:29.371932 14515 webserver.cc:460] Webserver started at http://127.28.235.195:43435/ using document root <none> and password file <none>
I20260926 07:23:29.372531 14515 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:29.372622 14515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:29.372859 14515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:29.375064 14515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "2149c6404f824ae1a8330ab422de4312"
format_stamp: "Formatted at 2026-09-26 07:23:29 on dist-test-slave-7zlw"
server_key: "3df408aa39dd5613006b7e3e54e5d630"
server_key_iv: "deb10a49539e9267f468692ae7d0cc31"
server_key_version: "encryptionkey@0"
I20260926 07:23:29.375651 14515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "2149c6404f824ae1a8330ab422de4312"
format_stamp: "Formatted at 2026-09-26 07:23:29 on dist-test-slave-7zlw"
server_key: "3df408aa39dd5613006b7e3e54e5d630"
server_key_iv: "deb10a49539e9267f468692ae7d0cc31"
server_key_version: "encryptionkey@0"
I20260926 07:23:29.379549 14515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:23:29.382251 14530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:29.383502 14515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:23:29.383647 14515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "2149c6404f824ae1a8330ab422de4312"
format_stamp: "Formatted at 2026-09-26 07:23:29 on dist-test-slave-7zlw"
server_key: "3df408aa39dd5613006b7e3e54e5d630"
server_key_iv: "deb10a49539e9267f468692ae7d0cc31"
server_key_version: "encryptionkey@0"
I20260926 07:23:29.383766 14515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:29.460083 14515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:29.463711 14515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:29.463919 14515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:29.464536 14515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:29.465453 14515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:29.465524 14515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:29.465610 14515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:29.465659 14515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:29.474968 14515 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:43273
I20260926 07:23:29.474987 14643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:43273 every 8 connection(s)
I20260926 07:23:29.475940 14515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:23:29.479286 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 14515
I20260926 07:23:29.479439 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:23:29.479734 29615 external_mini_cluster.cc:1428] Setting key 17de228013f77c392a4154147ecffc1a
Sep 26 07:23:29 dist-test-slave-7zlw krb5kdc[14140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407409, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:29.490233 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.477887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:41793 (local address 127.28.235.254:38195)
0926 07:23:29.478056 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:29.478061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:29.478912 (+ 851us) server_negotiation.cc:407] Connection header received
0926 07:23:29.480340 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:29.480343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:29.480392 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:29.480508 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:29.481844 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.482371 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.483521 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.483740 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.483862 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.486618 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:29.486622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:29.486624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:29.486697 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:29.488169 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:29.488655 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:29.488657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:29.488659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:29.488749 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:29.489164 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:29.489167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:29.489168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:29.489376 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:29.489472 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:29.489989 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 07:23:29.490096 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:23:29.491125 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.478301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:38195 (local address 127.28.235.195:41793)
0926 07:23:29.478750 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:29.478807 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:29.480103 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:29.480680 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:29.480687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:29.481192 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:29.481700 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.481714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.482500 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.482504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.483395 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.483404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.483899 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.483904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.484074 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.484693 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:29.484710 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:29.486437 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:29.488287 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:29.488292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:29.488304 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:29.488563 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:29.488842 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:29.488844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:29.488845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:29.489003 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:29.489479 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:29.489484 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:29.489734 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 07:23:29.489889 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 07:23:29.490176 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260926 07:23:29.492369 14644 heartbeater.cc:347] Connected to a master server at 127.28.235.254:38195
I20260926 07:23:29.492641 14644 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:29.493260 14644 heartbeater.cc:511] Master 127.28.235.254:38195 requested a full tablet report, sending...
I20260926 07:23:29.494284 14181 ts_manager.cc:194] Registered new tserver with Master: 2149c6404f824ae1a8330ab422de4312 (127.28.235.195:43273)
I20260926 07:23:29.494841 14181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:41793
I20260926 07:23:29.505743 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:23:29.515597 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.507837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.28.235.254:38195)
0926 07:23:29.508102 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:29.508107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:29.508260 (+ 153us) server_negotiation.cc:407] Connection header received
0926 07:23:29.508411 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:29.508421 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:29.508465 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:29.508549 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:29.509381 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.509930 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.510772 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.510994 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.511105 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.512502 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:29.512506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:29.512508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:29.512532 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:29.513806 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:29.514247 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:29.514251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:29.514253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:29.514380 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:29.514668 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:29.514672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:29.514674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:29.514903 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:29.515060 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:29.515307 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 07:23:29.515430 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:23:29.518851 14181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:23:29.522025 14181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:23:29.537315 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.531238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:35597 (local address 127.0.0.1:59744)
0926 07:23:29.531732 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:29.531780 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:29.531914 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:29.532533 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:29.532540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:29.532565 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:29.532713 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.532721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.534256 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.534260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.535605 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.535614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.536689 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.536693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.536812 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.536861 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 07:23:29.536985 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:23:29.537158 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260926 07:23:29.537319 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.531420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.28.235.194:35597)
0926 07:23:29.532029 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:29.532041 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:29.532068 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:23:29.532147 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:29.532152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:29.532370 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:29.532537 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:29.532864 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.534067 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.535752 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.536554 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.536679 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.536829 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:29.537050 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 07:23:29.537155 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":114,"threads_started":1}
I20260926 07:23:29.537669 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.531431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:43273 (local address 127.0.0.1:46156)
0926 07:23:29.532750 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:29.532766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:29.532851 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:29.533431 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:29.533434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:29.533462 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:29.533567 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.533577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.534754 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.534760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.536056 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.536068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.537244 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.537247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.537336 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.537364 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:23:29.537456 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:23:29.537529 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"mutex_wait_us":14,"thread_start_us":115,"threads_started":1}
I20260926 07:23:29.537966 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.531942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:46617 (local address 127.0.0.1:56676)
0926 07:23:29.532795 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:29.532808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:29.532899 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:29.533611 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:29.533616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:29.533638 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:29.533711 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.533718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.534907 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.534910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.536072 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.536078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.537562 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.537566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.537649 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.537671 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:23:29.537751 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:23:29.537834 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":95,"threads_started":1}
I20260926 07:23:29.538113 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.532018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.28.235.195:43273)
0926 07:23:29.532414 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:29.532427 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:29.533039 (+ 612us) server_negotiation.cc:407] Connection header received
0926 07:23:29.533119 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:29.533127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:29.533291 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:29.533452 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:29.533687 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.534617 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.536188 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.537002 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.537141 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.537256 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:29.537885 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 07:23:29.537985 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":172,"threads_started":1}
I20260926 07:23:29.538450 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.532375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.28.235.193:46617)
0926 07:23:29.533102 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:29.533110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:29.533135 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:23:29.533205 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:29.533210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:29.533387 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:29.533531 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:29.533837 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.534789 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.536224 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.536886 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.537025 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.537166 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:29.538200 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0926 07:23:29.538305 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":106,"threads_started":1}
I20260926 07:23:29.540637 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 52d9abd814234b93a44380059ab20617 (DEFAULT_TABLE table=test-table [id=57692572665b4233b353ec2345cbcbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:29.540675 14444 tablet_service.cc:1467] Processing CreateTablet for tablet 52d9abd814234b93a44380059ab20617 (DEFAULT_TABLE table=test-table [id=57692572665b4233b353ec2345cbcbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:29.540891 14310 tablet_service.cc:1467] Processing CreateTablet for tablet 52d9abd814234b93a44380059ab20617 (DEFAULT_TABLE table=test-table [id=57692572665b4233b353ec2345cbcbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:29.541721 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d9abd814234b93a44380059ab20617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:29.541733 14444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d9abd814234b93a44380059ab20617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:29.541744 14310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d9abd814234b93a44380059ab20617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:29.548908 14661 tablet_bootstrap.cc:492] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: Bootstrap starting.
I20260926 07:23:29.549311 14662 tablet_bootstrap.cc:492] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd: Bootstrap starting.
I20260926 07:23:29.549770 14663 tablet_bootstrap.cc:492] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: Bootstrap starting.
I20260926 07:23:29.551321 14661 tablet_bootstrap.cc:654] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:29.551474 14662 tablet_bootstrap.cc:654] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:29.551724 14663 tablet_bootstrap.cc:654] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:29.552503 14661 log.cc:824] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:29.552500 14662 log.cc:824] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:29.552981 14663 log.cc:824] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:29.554814 14662 tablet_bootstrap.cc:492] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd: No bootstrap required, opened a new log
I20260926 07:23:29.555012 14663 tablet_bootstrap.cc:492] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: No bootstrap required, opened a new log
I20260926 07:23:29.555027 14662 ts_tablet_manager.cc:1397] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 07:23:29.555042 14661 tablet_bootstrap.cc:492] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: No bootstrap required, opened a new log
I20260926 07:23:29.555179 14663 ts_tablet_manager.cc:1397] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 07:23:29.555274 14661 ts_tablet_manager.cc:1397] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 07:23:29.558320 14661 raft_consensus.cc:348] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 } } peers { permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 } } peers { permanent_uuid: "0b3d12a6aee74be7a221080d991ae8bd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46617 } }
I20260926 07:23:29.558601 14661 raft_consensus.cc:374] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:29.558663 14661 raft_consensus.cc:729] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2149c6404f824ae1a8330ab422de4312, State: Initialized, Role: FOLLOWER
I20260926 07:23:29.559145 14661 consensus_queue.cc:260] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 } } peers { permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 } } peers { permanent_uuid: "0b3d12a6aee74be7a221080d991ae8bd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46617 } }
I20260926 07:23:29.559106 14663 raft_consensus.cc:348] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 } } peers { permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 } } peers { permanent_uuid: "0b3d12a6aee74be7a221080d991ae8bd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46617 } }
I20260926 07:23:29.559356 14663 raft_consensus.cc:374] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:29.559407 14663 raft_consensus.cc:729] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0725e3ec3d4d9ebdd34f3b1cfe158f, State: Initialized, Role: FOLLOWER
I20260926 07:23:29.559593 14662 raft_consensus.cc:348] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 } } peers { permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 } } peers { permanent_uuid: "0b3d12a6aee74be7a221080d991ae8bd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46617 } }
I20260926 07:23:29.559821 14662 raft_consensus.cc:374] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:29.559872 14662 raft_consensus.cc:729] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3d12a6aee74be7a221080d991ae8bd, State: Initialized, Role: FOLLOWER
I20260926 07:23:29.559875 14644 heartbeater.cc:503] Master 127.28.235.254:38195 was elected leader, sending a full tablet report...
I20260926 07:23:29.559870 14663 consensus_queue.cc:260] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 } } peers { permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 } } peers { permanent_uuid: "0b3d12a6aee74be7a221080d991ae8bd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46617 } }
I20260926 07:23:29.560261 14661 ts_tablet_manager.cc:1428] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 07:23:29.560313 14662 consensus_queue.cc:260] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 } } peers { permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 } } peers { permanent_uuid: "0b3d12a6aee74be7a221080d991ae8bd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46617 } }
I20260926 07:23:29.561177 14662 ts_tablet_manager.cc:1428] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260926 07:23:29.561323 14663 ts_tablet_manager.cc:1428] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260926 07:23:29.561601 14376 heartbeater.cc:503] Master 127.28.235.254:38195 was elected leader, sending a full tablet report...
I20260926 07:23:29.561601 14510 heartbeater.cc:503] Master 127.28.235.254:38195 was elected leader, sending a full tablet report...
W20260926 07:23:29.680145 14511 tablet.cc:2369] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:23:29.697445 14377 tablet.cc:2369] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:23:29.727029 14645 tablet.cc:2369] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:23:29.766213 14669 raft_consensus.cc:482] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:29.766479 14669 raft_consensus.cc:504] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 } } peers { permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 } } peers { permanent_uuid: "0b3d12a6aee74be7a221080d991ae8bd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46617 } }
I20260926 07:23:29.767714 14669 leader_election.cc:288] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f0725e3ec3d4d9ebdd34f3b1cfe158f (127.28.235.194:35597), 2149c6404f824ae1a8330ab422de4312 (127.28.235.195:43273)
I20260926 07:23:29.772209 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.768091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:34447 (local address 127.28.235.195:43273)
0926 07:23:29.768218 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:29.768222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:29.768492 (+ 270us) server_negotiation.cc:407] Connection header received
0926 07:23:29.768638 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:29.768641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:29.768710 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:29.768787 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:29.769211 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.769772 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.771159 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.771592 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.771691 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.771774 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:29.771986 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 07:23:29.772067 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:23:29.772248 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.767899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:35597 (local address 127.28.235.193:50809)
0926 07:23:29.768311 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:29.768323 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:29.768418 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:29.768741 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:29.768747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:29.768768 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:29.769105 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.769117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.770066 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.770069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.771034 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.771044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.771782 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.771788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.771881 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.771898 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 07:23:29.772001 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 07:23:29.772066 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":38,"spinlock_wait_cycles":1792,"thread_start_us":145,"threads_started":1}
I20260926 07:23:29.772245 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.768021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:50809 (local address 127.28.235.194:35597)
0926 07:23:29.768177 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:29.768183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:29.768418 (+ 235us) server_negotiation.cc:407] Connection header received
0926 07:23:29.768559 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:29.768566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:29.768633 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:29.768707 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:29.769290 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.769950 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.771196 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.771622 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.771773 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.771832 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:29.772035 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 07:23:29.772124 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:23:29.772253 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.768003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:43273 (local address 127.28.235.193:34447)
0926 07:23:29.768347 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:29.768360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:29.768529 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:29.768823 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:29.768827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:29.768845 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:29.769072 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.769081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.769899 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.769903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.771022 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:29.771033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.771726 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.771731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:29.771830 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.771862 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:23:29.771942 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:23:29.772046 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":44,"thread_start_us":131,"threads_started":1}
I20260926 07:23:29.772989 14464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d9abd814234b93a44380059ab20617" candidate_uuid: "0b3d12a6aee74be7a221080d991ae8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" is_pre_election: true
I20260926 07:23:29.773096 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d9abd814234b93a44380059ab20617" candidate_uuid: "0b3d12a6aee74be7a221080d991ae8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2149c6404f824ae1a8330ab422de4312" is_pre_election: true
I20260926 07:23:29.773394 14464 raft_consensus.cc:2454] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b3d12a6aee74be7a221080d991ae8bd in term 0.
I20260926 07:23:29.773411 14598 raft_consensus.cc:2454] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b3d12a6aee74be7a221080d991ae8bd in term 0.
I20260926 07:23:29.773934 14265 leader_election.cc:302] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b3d12a6aee74be7a221080d991ae8bd, 2149c6404f824ae1a8330ab422de4312, 9f0725e3ec3d4d9ebdd34f3b1cfe158f; no voters:
I20260926 07:23:29.774225 14669 raft_consensus.cc:2781] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:23:29.774305 14669 raft_consensus.cc:482] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:23:29.774339 14669 raft_consensus.cc:3037] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:29.775489 14669 raft_consensus.cc:504] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 } } peers { permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 } } peers { permanent_uuid: "0b3d12a6aee74be7a221080d991ae8bd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46617 } }
I20260926 07:23:29.775911 14669 leader_election.cc:288] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [CANDIDATE]: Term 1 election: Requested vote from peers 9f0725e3ec3d4d9ebdd34f3b1cfe158f (127.28.235.194:35597), 2149c6404f824ae1a8330ab422de4312 (127.28.235.195:43273)
I20260926 07:23:29.776356 14464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d9abd814234b93a44380059ab20617" candidate_uuid: "0b3d12a6aee74be7a221080d991ae8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f"
I20260926 07:23:29.776356 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d9abd814234b93a44380059ab20617" candidate_uuid: "0b3d12a6aee74be7a221080d991ae8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2149c6404f824ae1a8330ab422de4312"
I20260926 07:23:29.776479 14464 raft_consensus.cc:3037] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:29.776513 14598 raft_consensus.cc:3037] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:29.777709 14464 raft_consensus.cc:2454] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b3d12a6aee74be7a221080d991ae8bd in term 1.
I20260926 07:23:29.777724 14598 raft_consensus.cc:2454] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b3d12a6aee74be7a221080d991ae8bd in term 1.
I20260926 07:23:29.778074 14265 leader_election.cc:302] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b3d12a6aee74be7a221080d991ae8bd, 9f0725e3ec3d4d9ebdd34f3b1cfe158f; no voters:
I20260926 07:23:29.778278 14669 raft_consensus.cc:2781] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:29.778534 14669 raft_consensus.cc:686] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [term 1 LEADER]: Becoming Leader. State: Replica: 0b3d12a6aee74be7a221080d991ae8bd, State: Running, Role: LEADER
I20260926 07:23:29.778985 14669 consensus_queue.cc:237] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 } } peers { permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 } } peers { permanent_uuid: "0b3d12a6aee74be7a221080d991ae8bd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46617 } }
I20260926 07:23:29.782609 14180 catalog_manager.cc:5495] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b3d12a6aee74be7a221080d991ae8bd (127.28.235.193). New cstate: current_term: 1 leader_uuid: "0b3d12a6aee74be7a221080d991ae8bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b3d12a6aee74be7a221080d991ae8bd" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 46617 } health_report { overall_health: HEALTHY } } }
I20260926 07:23:29.833766 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.829710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.28.235.193:46617)
0926 07:23:29.829872 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:29.829876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:29.829890 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:23:29.829975 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:29.829977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:29.830037 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:29.830142 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:23:29.830494 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.830993 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.831947 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:29.832167 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:29.832250 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:29.832778 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:23:29.833253 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:23:29.833375 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:29.833583 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 07:23:29.833649 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:23:29.840113 14464 raft_consensus.cc:1261] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f [term 1 FOLLOWER]: Refusing update from remote peer 0b3d12a6aee74be7a221080d991ae8bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:29.840113 14598 raft_consensus.cc:1261] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312 [term 1 FOLLOWER]: Refusing update from remote peer 0b3d12a6aee74be7a221080d991ae8bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:29.840943 14669 consensus_queue.cc:1035] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2149c6404f824ae1a8330ab422de4312" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:29.841272 14672 consensus_queue.cc:1035] T 52d9abd814234b93a44380059ab20617 P 0b3d12a6aee74be7a221080d991ae8bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0725e3ec3d4d9ebdd34f3b1cfe158f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 35597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:29.850775 14677 mvcc.cc:205] Tried to move back new op lower bound from 7333508750697279488 to 7333508750457950208. Current Snapshot: MvccSnapshot[applied={T|T < 7333508750697279488}]
I20260926 07:23:29.858839 14180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:29.859028 14180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:29.861698 14180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5990da145aee45d1b1085fefd8850ba1: Sending DeleteTablet for 3 replicas of tablet 52d9abd814234b93a44380059ab20617
I20260926 07:23:29.862596 14444 tablet_service.cc:1514] Processing DeleteTablet for tablet 52d9abd814234b93a44380059ab20617 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:29 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:59744
I20260926 07:23:29.862823 14310 tablet_service.cc:1514] Processing DeleteTablet for tablet 52d9abd814234b93a44380059ab20617 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:29 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:56676
I20260926 07:23:29.862901 14578 tablet_service.cc:1514] Processing DeleteTablet for tablet 52d9abd814234b93a44380059ab20617 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:29 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:46156
I20260926 07:23:29.863170 14685 tablet_replica.cc:330] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: stopping tablet replica
I20260926 07:23:29.863538 14685 raft_consensus.cc:2229] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:23:29.863581 14687 tablet_replica.cc:330] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: stopping tablet replica
I20260926 07:23:29.863907 14685 raft_consensus.cc:2258] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:29.864014 14687 raft_consensus.cc:2229] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:23:29.864224 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 14247
I20260926 07:23:29.864434 14687 raft_consensus.cc:2258] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:29.866307 14687 ts_tablet_manager.cc:1905] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:29.866858 14685 ts_tablet_manager.cc:1905] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:29.869321 14685 ts_tablet_manager.cc:1918] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:29.869442 14685 log.cc:1196] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/52d9abd814234b93a44380059ab20617
I20260926 07:23:29.869794 14685 ts_tablet_manager.cc:1939] T 52d9abd814234b93a44380059ab20617 P 9f0725e3ec3d4d9ebdd34f3b1cfe158f: Deleting consensus metadata
I20260926 07:23:29.869856 14687 ts_tablet_manager.cc:1918] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:29.869957 14687 log.cc:1196] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/52d9abd814234b93a44380059ab20617
I20260926 07:23:29.870301 14687 ts_tablet_manager.cc:1939] T 52d9abd814234b93a44380059ab20617 P 2149c6404f824ae1a8330ab422de4312: Deleting consensus metadata
I20260926 07:23:29.871192 14171 catalog_manager.cc:4841] TS 9f0725e3ec3d4d9ebdd34f3b1cfe158f (127.28.235.194:35597): tablet 52d9abd814234b93a44380059ab20617 (table test-table [id=57692572665b4233b353ec2345cbcbc8]) successfully deleted
I20260926 07:23:29.871513 14169 catalog_manager.cc:4841] TS 2149c6404f824ae1a8330ab422de4312 (127.28.235.195:43273): tablet 52d9abd814234b93a44380059ab20617 (table test-table [id=57692572665b4233b353ec2345cbcbc8]) successfully deleted
I20260926 07:23:29.872611 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 14381
W20260926 07:23:29.872673 14171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:46617 (error 108)
I20260926 07:23:29.877950 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:29.877577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:46617 (local address 127.0.0.1:56696)
0926 07:23:29.877737 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:29.877830 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:46617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 07:23:29.878186 14171 catalog_manager.cc:4584] TS 0b3d12a6aee74be7a221080d991ae8bd (127.28.235.193:46617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 52d9abd814234b93a44380059ab20617: Network error: Client connection negotiation failed: client connection to 127.28.235.193:46617: connect: Connection refused (error 111)
I20260926 07:23:29.879050 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 14515
I20260926 07:23:29.885193 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 14153
2026-09-26T07:23:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2685 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T07:23:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:29Z Disabled control of system clock
I20260926 07:23:29.916823 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:35301
--webserver_interface=127.28.235.254
--webserver_port=0
--builtin_ntp_servers=127.28.235.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:35301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:23:30.035087 14692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:30.035389 14692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:30.035429 14692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:30.039695 14692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:30.039764 14692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:30.039781 14692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:30.039798 14692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:30.039814 14692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:30.044924 14692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:35301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:35301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:30.046098 14692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:30.047566 14692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:30.056869 14700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:30.057039 14692 server_base.cc:1034] running on GCE node
W20260926 07:23:30.056861 14697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:30.057284 14698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:30.058066 14692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:30.059329 14692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:30.060565 14692 hybrid_clock.cc:648] HybridClock initialized: now 1790407410060552 us; error 29 us; skew 500 ppm
I20260926 07:23:30.062702 14692 webserver.cc:460] Webserver started at http://127.28.235.254:36973/ using document root <none> and password file <none>
I20260926 07:23:30.063303 14692 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:30.063405 14692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:30.063640 14692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:30.065850 14692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "05414ef2345d48c0b225205d7bb9e045"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "08e70945062d772da4771758be446b5c"
server_key_iv: "358400804cf25881c369cff7c3116677"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.066423 14692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "05414ef2345d48c0b225205d7bb9e045"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "08e70945062d772da4771758be446b5c"
server_key_iv: "358400804cf25881c369cff7c3116677"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.070283 14692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:23:30.072901 14706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:30.074266 14692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:30.074481 14692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "05414ef2345d48c0b225205d7bb9e045"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "08e70945062d772da4771758be446b5c"
server_key_iv: "358400804cf25881c369cff7c3116677"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.074602 14692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:30.090828 14692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:30.091616 14692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:30.091830 14692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:30.099162 14758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:35301 every 8 connection(s)
I20260926 07:23:30.099159 14692 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:35301
I20260926 07:23:30.100327 14692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:30.103706 14759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:30.103940 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 14692
I20260926 07:23:30.104074 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:30.104398 29615 external_mini_cluster.cc:1428] Setting key 22cd236f2c075d078e5d3d72946e4176
I20260926 07:23:30.110536 14759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045: Bootstrap starting.
I20260926 07:23:30.113560 14759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:30.114475 14759 log.cc:824] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:30.115020 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.105957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.28.235.254:35301)
0926 07:23:30.106721 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.106737 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.106776 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:23:30.107542 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.107586 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.108056 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.108367 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:30.108811 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.109974 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.111125 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.111703 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.111866 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.112930 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:30.112935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:30.112950 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:30.113486 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:30.113546 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:30.113566 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:30.113688 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.113892 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.114168 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":164,"threads_started":1}
I20260926 07:23:30.116743 14759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045: No bootstrap required, opened a new log
I20260926 07:23:30.120484 14759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05414ef2345d48c0b225205d7bb9e045" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35301 } }
I20260926 07:23:30.120790 14759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:30.120847 14759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05414ef2345d48c0b225205d7bb9e045, State: Initialized, Role: FOLLOWER
I20260926 07:23:30.121419 14759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05414ef2345d48c0b225205d7bb9e045" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35301 } }
I20260926 07:23:30.121611 14759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:30.121693 14759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:30.121771 14759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:30.122704 14759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05414ef2345d48c0b225205d7bb9e045" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35301 } }
I20260926 07:23:30.123018 14759 leader_election.cc:302] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05414ef2345d48c0b225205d7bb9e045; no voters:
I20260926 07:23:30.123287 14759 leader_election.cc:288] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:30.123431 14764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:30.123692 14764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [term 1 LEADER]: Becoming Leader. State: Replica: 05414ef2345d48c0b225205d7bb9e045, State: Running, Role: LEADER
I20260926 07:23:30.124017 14764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05414ef2345d48c0b225205d7bb9e045" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35301 } }
I20260926 07:23:30.124621 14759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:30.125753 14766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05414ef2345d48c0b225205d7bb9e045. Latest consensus state: current_term: 1 leader_uuid: "05414ef2345d48c0b225205d7bb9e045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05414ef2345d48c0b225205d7bb9e045" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35301 } } }
I20260926 07:23:30.125790 14765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05414ef2345d48c0b225205d7bb9e045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05414ef2345d48c0b225205d7bb9e045" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35301 } } }
I20260926 07:23:30.125896 14766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:30.125911 14765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:30.126559 14773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:30.129694 14773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:30.136479 14773 catalog_manager.cc:1347] Generated new cluster ID: 13ad39df3ad242a9b35db759b954b924
I20260926 07:23:30.136582 14773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:30.152302 14773 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:30.153738 14773 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:30.162035 14773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045: Generated new TSK 0
I20260926 07:23:30.163041 14773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:30.166117 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35301
--builtin_ntp_servers=127.28.235.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:23:30.282344 14783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:30.282666 14783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:30.282752 14783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:30.288070 14783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:30.288170 14783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:30.288268 14783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:23:30.293519 14783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:30.294673 14783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:30.296084 14783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:30.306048 14789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:30.306095 14791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:30.306214 14783 server_base.cc:1034] running on GCE node
W20260926 07:23:30.306048 14788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:30.307008 14783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:30.307677 14783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:30.308902 14783 hybrid_clock.cc:648] HybridClock initialized: now 1790407410308883 us; error 30 us; skew 500 ppm
I20260926 07:23:30.311066 14783 webserver.cc:460] Webserver started at http://127.28.235.193:40707/ using document root <none> and password file <none>
I20260926 07:23:30.311712 14783 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:30.311786 14783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:30.312049 14783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:30.314440 14783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "fbec8a4212044b7faf190fcc5afd60bf"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "353c6b772f6b0e9f9a8e178ccdb1e902"
server_key_iv: "0e75e0335ae39397a9d6ae678b74f418"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.315083 14783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "fbec8a4212044b7faf190fcc5afd60bf"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "353c6b772f6b0e9f9a8e178ccdb1e902"
server_key_iv: "0e75e0335ae39397a9d6ae678b74f418"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.319504 14783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:23:30.322461 14797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:30.323985 14783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:23:30.324139 14783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "fbec8a4212044b7faf190fcc5afd60bf"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "353c6b772f6b0e9f9a8e178ccdb1e902"
server_key_iv: "0e75e0335ae39397a9d6ae678b74f418"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.324275 14783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:30.335146 14783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:30.335806 14783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:30.335955 14783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:30.336500 14783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:30.337585 14783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:30.337634 14783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:30.337677 14783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:30.337692 14783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:30.347527 14783 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:41545
I20260926 07:23:30.347545 14910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:41545 every 8 connection(s)
I20260926 07:23:30.348497 14783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:23:30.353526 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 14783
I20260926 07:23:30.353696 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:23:30.354018 29615 external_mini_cluster.cc:1428] Setting key 1f16415d054124b5b0a43da6e79bc328
I20260926 07:23:30.356300 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35301
--builtin_ntp_servers=127.28.235.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:23:30.360351 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.350531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:37895 (local address 127.28.235.254:35301)
0926 07:23:30.350692 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.350696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.351869 (+ 1173us) server_negotiation.cc:407] Connection header received
0926 07:23:30.353442 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.353445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.353497 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.353578 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:30.354909 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.355752 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.357770 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.358190 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.358416 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.359675 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:30.359681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:30.359688 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:30.359707 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:30.359734 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:30.359747 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:30.359828 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.359988 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.360069 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:23:30.361235 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.351009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:35301 (local address 127.28.235.193:37895)
0926 07:23:30.351715 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:30.351756 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:30.353156 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:30.353822 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:30.353833 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:30.353962 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:30.354690 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.354709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.355903 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.355906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.357584 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.357601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.358458 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.358462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.358616 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.359373 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:23:30.359375 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:30.359421 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:23:30.359498 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:30.359839 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:30.359846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:30.359850 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:23:30.359955 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:23:30.360206 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":127,"threads_started":1}
I20260926 07:23:30.362622 14911 heartbeater.cc:347] Connected to a master server at 127.28.235.254:35301
I20260926 07:23:30.362861 14911 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:30.363466 14911 heartbeater.cc:511] Master 127.28.235.254:35301 requested a full tablet report, sending...
I20260926 07:23:30.365311 14723 ts_manager.cc:194] Registered new tserver with Master: fbec8a4212044b7faf190fcc5afd60bf (127.28.235.193:41545)
I20260926 07:23:30.366489 14723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.193:37895
W20260926 07:23:30.470191 14914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:30.470464 14914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:30.470503 14914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:30.474647 14914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:30.474711 14914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:30.474767 14914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:23:30.479820 14914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:30.480988 14914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:30.482527 14914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:30.492838 14922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:30.492908 14914 server_base.cc:1034] running on GCE node
W20260926 07:23:30.492835 14919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:30.492849 14920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:30.493490 14914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:30.494189 14914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:30.495432 14914 hybrid_clock.cc:648] HybridClock initialized: now 1790407410495403 us; error 44 us; skew 500 ppm
I20260926 07:23:30.497570 14914 webserver.cc:460] Webserver started at http://127.28.235.194:45763/ using document root <none> and password file <none>
I20260926 07:23:30.498188 14914 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:30.498286 14914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:30.498551 14914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:30.500702 14914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "c87ba713dfff4556aa9ae3d128dc5c5c"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "5177b49ada4b6631a4a1e3af220f1e43"
server_key_iv: "76479142036e2288b370a3828642d4fc"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.501319 14914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "c87ba713dfff4556aa9ae3d128dc5c5c"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "5177b49ada4b6631a4a1e3af220f1e43"
server_key_iv: "76479142036e2288b370a3828642d4fc"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.506187 14914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 07:23:30.508978 14928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:30.510354 14914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:23:30.510535 14914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "c87ba713dfff4556aa9ae3d128dc5c5c"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "5177b49ada4b6631a4a1e3af220f1e43"
server_key_iv: "76479142036e2288b370a3828642d4fc"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.510666 14914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:30.533099 14914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:30.534175 14914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:30.534406 14914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:30.535073 14914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:30.536080 14914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:30.536151 14914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:30.536222 14914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:30.536274 14914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:30.545584 14914 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:33049
I20260926 07:23:30.545607 15041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:33049 every 8 connection(s)
I20260926 07:23:30.546540 14914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:23:30.554059 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 14914
I20260926 07:23:30.554229 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:23:30.554530 29615 external_mini_cluster.cc:1428] Setting key 7b5d9eb0f0614c1b8e8bc98508253469
I20260926 07:23:30.557154 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.548700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:47263 (local address 127.28.235.254:35301)
0926 07:23:30.548887 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.548891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.550007 (+ 1116us) server_negotiation.cc:407] Connection header received
0926 07:23:30.551388 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.551390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.551447 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.551527 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:30.552612 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.553172 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.554630 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.554933 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.555037 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.556514 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:30.556517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:30.556520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:30.556533 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:30.556557 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:30.556568 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:30.556647 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.556868 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.556947 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 07:23:30.557174 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35301
--builtin_ntp_servers=127.28.235.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:23:30.558099 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.549164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:35301 (local address 127.28.235.194:47263)
0926 07:23:30.549859 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:30.549898 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:30.551190 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:30.551747 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:30.551755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:30.551844 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:30.552439 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.552454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.553346 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.553350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.554464 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.554476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.555231 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.555237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.555465 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.556171 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:23:30.556173 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:30.556235 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:23:30.556328 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:30.556680 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:30.556688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:30.556693 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:23:30.556834 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 07:23:30.557171 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
I20260926 07:23:30.559382 15042 heartbeater.cc:347] Connected to a master server at 127.28.235.254:35301
I20260926 07:23:30.559626 15042 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:30.560189 15042 heartbeater.cc:511] Master 127.28.235.254:35301 requested a full tablet report, sending...
I20260926 07:23:30.561349 14723 ts_manager.cc:194] Registered new tserver with Master: c87ba713dfff4556aa9ae3d128dc5c5c (127.28.235.194:33049)
I20260926 07:23:30.561928 14723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.194:47263
W20260926 07:23:30.669368 15045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:30.669636 15045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:30.669756 15045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:30.673851 15045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:30.673908 15045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:30.674010 15045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:23:30.679011 15045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:35301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:30.680117 15045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:30.681501 15045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:30.691399 15050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:30.691399 15051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:30.691399 15053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:30.691710 15045 server_base.cc:1034] running on GCE node
I20260926 07:23:30.692075 15045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:30.692715 15045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:30.693924 15045 hybrid_clock.cc:648] HybridClock initialized: now 1790407410693868 us; error 65 us; skew 500 ppm
I20260926 07:23:30.695920 15045 webserver.cc:460] Webserver started at http://127.28.235.195:43449/ using document root <none> and password file <none>
I20260926 07:23:30.696550 15045 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:30.696647 15045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:30.696882 15045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:30.699106 15045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "93fdd2d0f1044625b9d7111638cf48e0"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "94b69daad1197fc98f2892a14f2dc1ce"
server_key_iv: "568aab674030e2492b67c750f17e2961"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.699690 15045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "93fdd2d0f1044625b9d7111638cf48e0"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "94b69daad1197fc98f2892a14f2dc1ce"
server_key_iv: "568aab674030e2492b67c750f17e2961"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.703568 15045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:23:30.706096 15059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:30.707324 15045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:23:30.707508 15045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "93fdd2d0f1044625b9d7111638cf48e0"
format_stamp: "Formatted at 2026-09-26 07:23:30 on dist-test-slave-7zlw"
server_key: "94b69daad1197fc98f2892a14f2dc1ce"
server_key_iv: "568aab674030e2492b67c750f17e2961"
server_key_version: "encryptionkey@0"
I20260926 07:23:30.707638 15045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:30.729862 15045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:30.730607 15045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:30.730814 15045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:30.731417 15045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:30.732401 15045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:30.732481 15045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:30.732558 15045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:30.732602 15045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:30.742000 15045 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:37985
I20260926 07:23:30.742027 15172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:37985 every 8 connection(s)
I20260926 07:23:30.742986 15045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:23:30.743993 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 15045
I20260926 07:23:30.744079 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:23:30.744355 29615 external_mini_cluster.cc:1428] Setting key be9cb780fb3355e3a502b88b6507ebe4
I20260926 07:23:30.752792 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.745045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:37291 (local address 127.28.235.254:35301)
0926 07:23:30.745175 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.745179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.746597 (+ 1418us) server_negotiation.cc:407] Connection header received
0926 07:23:30.748048 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.748052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.748106 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.748183 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:30.749179 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.749714 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.750847 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.751081 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.751193 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.752300 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:30.752304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:30.752307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:30.752319 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:30.752347 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:30.752355 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:30.752432 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.752590 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.752649 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:23:30.753702 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.745684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:35301 (local address 127.28.235.195:37291)
0926 07:23:30.746393 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:30.746497 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:30.747841 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:30.748377 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:30.748386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:30.748492 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:30.748981 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.748994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.749838 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.749841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.750719 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.750728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.751230 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.751233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.751396 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.752037 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:23:30.752039 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:30.752077 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:23:30.752139 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:30.752443 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:30.752453 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:30.752465 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 07:23:30.752590 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 07:23:30.752862 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":166,"threads_started":1}
I20260926 07:23:30.754840 15173 heartbeater.cc:347] Connected to a master server at 127.28.235.254:35301
I20260926 07:23:30.755092 15173 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:30.755623 15173 heartbeater.cc:511] Master 127.28.235.254:35301 requested a full tablet report, sending...
I20260926 07:23:30.756716 14723 ts_manager.cc:194] Registered new tserver with Master: 93fdd2d0f1044625b9d7111638cf48e0 (127.28.235.195:37985)
I20260926 07:23:30.757550 14723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.195:37291
I20260926 07:23:30.758574 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:23:30.764806 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.760875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.28.235.254:35301)
0926 07:23:30.760999 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.761003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.761316 (+ 313us) server_negotiation.cc:407] Connection header received
0926 07:23:30.761426 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.761445 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.761485 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.761555 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:30.761816 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.762308 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.763213 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.763481 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.763571 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.764427 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:30.764429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:30.764432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:30.764442 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:30.764458 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:30.764465 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:30.764524 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.764653 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.764701 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:23:30.767758 14723 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:23:30.771083 14723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:23:30.785594 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.779808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:33049 (local address 127.0.0.1:45044)
0926 07:23:30.780618 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:30.780665 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:30.780809 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:30.781627 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:30.781633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:30.781662 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:30.781789 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.781794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.782903 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.782906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.784078 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.784087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.785046 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.785051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.785152 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.785188 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:23:30.785297 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:23:30.785429 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260926 07:23:30.785601 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.779928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.28.235.194:33049)
0926 07:23:30.780665 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.780677 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.780979 (+ 302us) server_negotiation.cc:407] Connection header received
0926 07:23:30.781088 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.781094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.781301 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.781453 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:30.781905 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.782779 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.784214 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.784893 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.785047 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.785227 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.785335 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.785429 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":112,"threads_started":1}
I20260926 07:23:30.786078 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.779819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:41545 (local address 127.0.0.1:48802)
0926 07:23:30.780839 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:30.780852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:30.780939 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:30.781653 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:30.781655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:30.781673 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:30.781852 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.781859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.783333 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.783337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.784376 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.784384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.785670 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.785674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.785790 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.785815 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:23:30.785882 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:23:30.785945 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":60,"spinlock_wait_cycles":5504,"thread_start_us":107,"threads_started":1}
I20260926 07:23:30.786365 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.780515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:37985 (local address 127.0.0.1:43472)
0926 07:23:30.781073 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:30.781086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:30.781163 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:30.782002 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:30.782005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:30.782023 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:30.782093 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.782098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.783380 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.783383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.784492 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.784501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.785670 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.785674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.786067 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.786091 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:23:30.786165 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:23:30.786237 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":298,"thread_start_us":84,"threads_started":1}
I20260926 07:23:30.786517 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.780106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.28.235.193:41545)
0926 07:23:30.780872 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.780889 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.780917 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:23:30.781126 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.781133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.781333 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.781468 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:30.781993 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.783173 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.784497 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.785324 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.785451 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.785602 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.786251 (+ 649us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.786342 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":136,"threads_started":1}
I20260926 07:23:30.786841 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.780691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.28.235.195:37985)
0926 07:23:30.781427 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.781440 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.781462 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:23:30.781669 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.781675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.781865 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.782011 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:30.782186 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.783259 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.784606 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.785363 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.785463 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.785611 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.786579 (+ 968us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.786685 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":99,"threads_started":1}
I20260926 07:23:30.788929 14976 tablet_service.cc:1467] Processing CreateTablet for tablet 46cedf1b106642799ebdb1aad5232e33 (DEFAULT_TABLE table=test-table [id=cf128c6cdb004fa9bd1ea868cd9ecf00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:30.789079 14845 tablet_service.cc:1467] Processing CreateTablet for tablet 46cedf1b106642799ebdb1aad5232e33 (DEFAULT_TABLE table=test-table [id=cf128c6cdb004fa9bd1ea868cd9ecf00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:30.789901 14845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cedf1b106642799ebdb1aad5232e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:30.789901 14976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cedf1b106642799ebdb1aad5232e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:30.793745 15107 tablet_service.cc:1467] Processing CreateTablet for tablet 46cedf1b106642799ebdb1aad5232e33 (DEFAULT_TABLE table=test-table [id=cf128c6cdb004fa9bd1ea868cd9ecf00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:30.794821 15107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cedf1b106642799ebdb1aad5232e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:30.796087 15190 tablet_bootstrap.cc:492] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: Bootstrap starting.
I20260926 07:23:30.797278 15191 tablet_bootstrap.cc:492] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf: Bootstrap starting.
I20260926 07:23:30.798926 15190 tablet_bootstrap.cc:654] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:30.799348 15191 tablet_bootstrap.cc:654] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:30.799716 15190 log.cc:824] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:30.800139 15191 log.cc:824] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:30.801663 15190 tablet_bootstrap.cc:492] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: No bootstrap required, opened a new log
I20260926 07:23:30.801968 15194 tablet_bootstrap.cc:492] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: Bootstrap starting.
I20260926 07:23:30.802585 15190 ts_tablet_manager.cc:1397] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:23:30.802639 15191 tablet_bootstrap.cc:492] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf: No bootstrap required, opened a new log
I20260926 07:23:30.802829 15191 ts_tablet_manager.cc:1397] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:23:30.803911 15194 tablet_bootstrap.cc:654] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:30.804543 15194 log.cc:824] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:30.806099 15194 tablet_bootstrap.cc:492] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: No bootstrap required, opened a new log
I20260926 07:23:30.806283 15194 ts_tablet_manager.cc:1397] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 07:23:30.806877 15191 raft_consensus.cc:348] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 } } peers { permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 } } peers { permanent_uuid: "93fdd2d0f1044625b9d7111638cf48e0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 37985 } }
I20260926 07:23:30.806877 15190 raft_consensus.cc:348] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 } } peers { permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 } } peers { permanent_uuid: "93fdd2d0f1044625b9d7111638cf48e0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 37985 } }
I20260926 07:23:30.807139 15191 raft_consensus.cc:374] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:30.807139 15190 raft_consensus.cc:374] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:30.807201 15190 raft_consensus.cc:729] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87ba713dfff4556aa9ae3d128dc5c5c, State: Initialized, Role: FOLLOWER
I20260926 07:23:30.807227 15191 raft_consensus.cc:729] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbec8a4212044b7faf190fcc5afd60bf, State: Initialized, Role: FOLLOWER
I20260926 07:23:30.807706 15191 consensus_queue.cc:260] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 } } peers { permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 } } peers { permanent_uuid: "93fdd2d0f1044625b9d7111638cf48e0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 37985 } }
I20260926 07:23:30.807706 15190 consensus_queue.cc:260] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 } } peers { permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 } } peers { permanent_uuid: "93fdd2d0f1044625b9d7111638cf48e0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 37985 } }
I20260926 07:23:30.808578 15042 heartbeater.cc:503] Master 127.28.235.254:35301 was elected leader, sending a full tablet report...
I20260926 07:23:30.808693 15190 ts_tablet_manager.cc:1428] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:23:30.808755 15191 ts_tablet_manager.cc:1428] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:23:30.809134 14911 heartbeater.cc:503] Master 127.28.235.254:35301 was elected leader, sending a full tablet report...
I20260926 07:23:30.810314 15194 raft_consensus.cc:348] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 } } peers { permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 } } peers { permanent_uuid: "93fdd2d0f1044625b9d7111638cf48e0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 37985 } }
I20260926 07:23:30.810563 15194 raft_consensus.cc:374] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:30.810621 15194 raft_consensus.cc:729] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93fdd2d0f1044625b9d7111638cf48e0, State: Initialized, Role: FOLLOWER
I20260926 07:23:30.811053 15194 consensus_queue.cc:260] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 } } peers { permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 } } peers { permanent_uuid: "93fdd2d0f1044625b9d7111638cf48e0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 37985 } }
I20260926 07:23:30.811717 15173 heartbeater.cc:503] Master 127.28.235.254:35301 was elected leader, sending a full tablet report...
I20260926 07:23:30.812083 15194 ts_tablet_manager.cc:1428] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 07:23:30.849802 14912 tablet.cc:2369] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:23:30.927951 15198 raft_consensus.cc:482] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:30.928169 15198 raft_consensus.cc:504] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 } } peers { permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 } } peers { permanent_uuid: "93fdd2d0f1044625b9d7111638cf48e0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 37985 } }
I20260926 07:23:30.929548 15198 leader_election.cc:288] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87ba713dfff4556aa9ae3d128dc5c5c (127.28.235.194:33049), fbec8a4212044b7faf190fcc5afd60bf (127.28.235.193:41545)
I20260926 07:23:30.933696 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.929869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:38685 (local address 127.28.235.194:33049)
0926 07:23:30.929991 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.929996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.930075 (+ 79us) server_negotiation.cc:407] Connection header received
0926 07:23:30.930212 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.930214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.930284 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.930394 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:30.930712 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.931390 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.932643 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.933096 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.933199 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.933270 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.933477 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.933562 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:23:30.933699 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.929777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:33049 (local address 127.28.235.195:38685)
0926 07:23:30.929973 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:30.929990 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:30.930100 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:30.930407 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:30.930410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:30.930437 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:30.930572 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.930579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.931560 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.931564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.932518 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.932528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.933224 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.933231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.933333 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.933368 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:23:30.933446 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:23:30.933542 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":57}
I20260926 07:23:30.933729 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.929894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:44069 (local address 127.28.235.193:41545)
0926 07:23:30.930019 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.930026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.930292 (+ 266us) server_negotiation.cc:407] Connection header received
0926 07:23:30.930412 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.930416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.930463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.930549 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:30.930870 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.931483 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.932686 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.933173 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.933289 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.933338 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.933544 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.933620 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:23:30.933709 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.929803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:41545 (local address 127.28.235.195:44069)
0926 07:23:30.930197 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:30.930216 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:30.930302 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:30.930578 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:30.930580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:30.930596 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:30.930730 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.930737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.931633 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.931636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.932528 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:30.932538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.933320 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.933323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:30.933394 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.933410 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:23:30.933507 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:23:30.933573 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":50,"thread_start_us":118,"threads_started":1}
I20260926 07:23:30.934477 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cedf1b106642799ebdb1aad5232e33" candidate_uuid: "93fdd2d0f1044625b9d7111638cf48e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" is_pre_election: true
I20260926 07:23:30.934764 14865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cedf1b106642799ebdb1aad5232e33" candidate_uuid: "93fdd2d0f1044625b9d7111638cf48e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbec8a4212044b7faf190fcc5afd60bf" is_pre_election: true
I20260926 07:23:30.934890 14996 raft_consensus.cc:2454] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93fdd2d0f1044625b9d7111638cf48e0 in term 0.
I20260926 07:23:30.935031 14865 raft_consensus.cc:2454] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93fdd2d0f1044625b9d7111638cf48e0 in term 0.
I20260926 07:23:30.935412 15062 leader_election.cc:302] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93fdd2d0f1044625b9d7111638cf48e0, c87ba713dfff4556aa9ae3d128dc5c5c; no voters:
I20260926 07:23:30.935726 15198 raft_consensus.cc:2781] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:23:30.935823 15198 raft_consensus.cc:482] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:23:30.935879 15198 raft_consensus.cc:3037] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:30.936926 15198 raft_consensus.cc:504] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 } } peers { permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 } } peers { permanent_uuid: "93fdd2d0f1044625b9d7111638cf48e0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 37985 } }
I20260926 07:23:30.937335 15198 leader_election.cc:288] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [CANDIDATE]: Term 1 election: Requested vote from peers c87ba713dfff4556aa9ae3d128dc5c5c (127.28.235.194:33049), fbec8a4212044b7faf190fcc5afd60bf (127.28.235.193:41545)
I20260926 07:23:30.937675 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cedf1b106642799ebdb1aad5232e33" candidate_uuid: "93fdd2d0f1044625b9d7111638cf48e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c"
I20260926 07:23:30.937690 14865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cedf1b106642799ebdb1aad5232e33" candidate_uuid: "93fdd2d0f1044625b9d7111638cf48e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbec8a4212044b7faf190fcc5afd60bf"
I20260926 07:23:30.937819 14996 raft_consensus.cc:3037] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:30.937846 14865 raft_consensus.cc:3037] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:30.939033 14996 raft_consensus.cc:2454] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93fdd2d0f1044625b9d7111638cf48e0 in term 1.
I20260926 07:23:30.939034 14865 raft_consensus.cc:2454] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93fdd2d0f1044625b9d7111638cf48e0 in term 1.
I20260926 07:23:30.939342 15062 leader_election.cc:302] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93fdd2d0f1044625b9d7111638cf48e0, c87ba713dfff4556aa9ae3d128dc5c5c; no voters:
I20260926 07:23:30.939564 15198 raft_consensus.cc:2781] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:30.939742 15198 raft_consensus.cc:686] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 1 LEADER]: Becoming Leader. State: Replica: 93fdd2d0f1044625b9d7111638cf48e0, State: Running, Role: LEADER
I20260926 07:23:30.940106 15198 consensus_queue.cc:237] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 } } peers { permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 } } peers { permanent_uuid: "93fdd2d0f1044625b9d7111638cf48e0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 37985 } }
I20260926 07:23:30.943185 14721 catalog_manager.cc:5495] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93fdd2d0f1044625b9d7111638cf48e0 (127.28.235.195). New cstate: current_term: 1 leader_uuid: "93fdd2d0f1044625b9d7111638cf48e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93fdd2d0f1044625b9d7111638cf48e0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 37985 } health_report { overall_health: HEALTHY } } }
I20260926 07:23:30.976766 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:30.973425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.28.235.195:37985)
0926 07:23:30.973556 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:30.973560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:30.973573 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:23:30.973688 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:30.973690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:30.973743 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:30.973830 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:23:30.974081 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.974573 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.975450 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:30.975653 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:30.975728 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:30.975976 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:23:30.976394 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:23:30.976486 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:30.976541 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 07:23:30.976585 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:23:30.982681 14865 raft_consensus.cc:1261] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf [term 1 FOLLOWER]: Refusing update from remote peer 93fdd2d0f1044625b9d7111638cf48e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:30.982681 14996 raft_consensus.cc:1261] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c [term 1 FOLLOWER]: Refusing update from remote peer 93fdd2d0f1044625b9d7111638cf48e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:30.983398 15198 consensus_queue.cc:1035] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbec8a4212044b7faf190fcc5afd60bf" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:30.983572 15200 consensus_queue.cc:1035] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87ba713dfff4556aa9ae3d128dc5c5c" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 33049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:30.991533 15203 mvcc.cc:205] Tried to move back new op lower bound from 7333508755377930240 to 7333508755213058048. Current Snapshot: MvccSnapshot[applied={T|T < 7333508755377930240}]
I20260926 07:23:30.993345 15204 mvcc.cc:205] Tried to move back new op lower bound from 7333508755377930240 to 7333508755213058048. Current Snapshot: MvccSnapshot[applied={T|T < 7333508755377930240}]
I20260926 07:23:30.994479 15205 mvcc.cc:205] Tried to move back new op lower bound from 7333508755377930240 to 7333508755213058048. Current Snapshot: MvccSnapshot[applied={T|T < 7333508755377930240}]
I20260926 07:23:31.001083 14723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:31.001282 14723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:31.004316 14723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05414ef2345d48c0b225205d7bb9e045: Sending DeleteTablet for 3 replicas of tablet 46cedf1b106642799ebdb1aad5232e33
I20260926 07:23:31.005275 14976 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cedf1b106642799ebdb1aad5232e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:31 UTC) from {username='slave'} at 127.0.0.1:45044
I20260926 07:23:31.005455 14845 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cedf1b106642799ebdb1aad5232e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:31 UTC) from {username='slave'} at 127.0.0.1:48802
I20260926 07:23:31.005821 15212 tablet_replica.cc:330] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: stopping tablet replica
I20260926 07:23:31.005926 15107 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cedf1b106642799ebdb1aad5232e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:31 UTC) from {username='slave'} at 127.0.0.1:43472
I20260926 07:23:31.006171 15212 raft_consensus.cc:2229] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:23:31.006418 15213 tablet_replica.cc:330] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf: stopping tablet replica
I20260926 07:23:31.006497 15212 raft_consensus.cc:2258] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:31.006705 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 14783
I20260926 07:23:31.006718 15213 raft_consensus.cc:2229] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:23:31.007019 15213 raft_consensus.cc:2258] T 46cedf1b106642799ebdb1aad5232e33 P fbec8a4212044b7faf190fcc5afd60bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:31.009749 15212 ts_tablet_manager.cc:1905] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:31.013439 15212 ts_tablet_manager.cc:1918] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:31.013554 15212 log.cc:1196] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/46cedf1b106642799ebdb1aad5232e33
I20260926 07:23:31.013917 15212 ts_tablet_manager.cc:1939] T 46cedf1b106642799ebdb1aad5232e33 P c87ba713dfff4556aa9ae3d128dc5c5c: Deleting consensus metadata
I20260926 07:23:31.014885 15214 tablet_replica.cc:330] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: stopping tablet replica
I20260926 07:23:31.015295 15214 raft_consensus.cc:2229] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:23:31.015329 14709 catalog_manager.cc:4841] TS c87ba713dfff4556aa9ae3d128dc5c5c (127.28.235.194:33049): tablet 46cedf1b106642799ebdb1aad5232e33 (table test-table [id=cf128c6cdb004fa9bd1ea868cd9ecf00]) successfully deleted
I20260926 07:23:31.015725 15214 raft_consensus.cc:2258] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 07:23:31.016924 14707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:41545 (error 108)
I20260926 07:23:31.017145 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 14914
I20260926 07:23:31.018544 15214 ts_tablet_manager.cc:1905] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:31.018589 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:31.018115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:41545 (local address 127.0.0.1:48816)
0926 07:23:31.018410 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:31.018490 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:41545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
W20260926 07:23:31.018762 14707 catalog_manager.cc:4584] TS fbec8a4212044b7faf190fcc5afd60bf (127.28.235.193:41545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46cedf1b106642799ebdb1aad5232e33: Network error: Client connection negotiation failed: client connection to 127.28.235.193:41545: connect: Connection refused (error 111)
I20260926 07:23:31.021766 15214 ts_tablet_manager.cc:1918] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:31.021924 15214 log.cc:1196] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/46cedf1b106642799ebdb1aad5232e33
I20260926 07:23:31.022320 15214 ts_tablet_manager.cc:1939] T 46cedf1b106642799ebdb1aad5232e33 P 93fdd2d0f1044625b9d7111638cf48e0: Deleting consensus metadata
I20260926 07:23:31.023136 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 15045
I20260926 07:23:31.029825 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 14692
2026-09-26T07:23:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1143 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:31 dist-test-slave-7zlw krb5kdc[15218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:31 dist-test-slave-7zlw krb5kdc[15218](info): set up 2 sockets
Sep 26 07:23:31 dist-test-slave-7zlw krb5kdc[15218](info): commencing operation
krb5kdc: starting...
W20260926 07:23:32.072975 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:32 dist-test-slave-7zlw krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:23:32.204079 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:45821
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:45821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:32.319967 15231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:32.320315 15231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:32.320401 15231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:32.324692 15231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:32.324810 15231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:32.324851 15231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:32.324889 15231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:32.324924 15231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:32.330727 15231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:45821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:45821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:32.331914 15231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:32.333069 15231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:32.342479 15237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:32.342499 15239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:32.342545 15236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:32.342824 15231 server_base.cc:1034] running on GCE node
I20260926 07:23:32.343472 15231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:32.344583 15231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:32.345782 15231 hybrid_clock.cc:648] HybridClock initialized: now 1790407412345716 us; error 82 us; skew 500 ppm
Sep 26 07:23:32 dist-test-slave-7zlw krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407412, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:32.353165 15231 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:23:32.354171 15231 webserver.cc:460] Webserver started at http://127.28.235.254:37767/ using document root <none> and password file <none>
I20260926 07:23:32.354822 15231 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:32.354920 15231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:32.355158 15231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:32.357410 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "6666c0b735944087827897f0ca9ffbff"
format_stamp: "Formatted at 2026-09-26 07:23:32 on dist-test-slave-7zlw"
server_key: "691ab481496881028ff92f075a25664a"
server_key_iv: "e4d0971f1acf9f2c52dca2360b595316"
server_key_version: "encryptionkey@0"
I20260926 07:23:32.358009 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "6666c0b735944087827897f0ca9ffbff"
format_stamp: "Formatted at 2026-09-26 07:23:32 on dist-test-slave-7zlw"
server_key: "691ab481496881028ff92f075a25664a"
server_key_iv: "e4d0971f1acf9f2c52dca2360b595316"
server_key_version: "encryptionkey@0"
I20260926 07:23:32.362056 15231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:23:32.364908 15246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:32.366340 15231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:23:32.366508 15231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "6666c0b735944087827897f0ca9ffbff"
format_stamp: "Formatted at 2026-09-26 07:23:32 on dist-test-slave-7zlw"
server_key: "691ab481496881028ff92f075a25664a"
server_key_iv: "e4d0971f1acf9f2c52dca2360b595316"
server_key_version: "encryptionkey@0"
I20260926 07:23:32.366639 15231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:32.383819 15231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:32.387164 15231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:32.387385 15231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:32.394647 15231 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:45821
I20260926 07:23:32.394670 15298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:45821 every 8 connection(s)
I20260926 07:23:32.395664 15231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:32.398542 15299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:32.401454 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 15231
I20260926 07:23:32.401575 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:32.401924 29615 external_mini_cluster.cc:1428] Setting key 43309eab6342ab28a5d3052d700f4c60
I20260926 07:23:32.404876 15299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff: Bootstrap starting.
I20260926 07:23:32.407655 15299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:32.408468 15299 log.cc:824] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:32.410825 15299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff: No bootstrap required, opened a new log
Sep 26 07:23:32 dist-test-slave-7zlw krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:32.415386 15299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c0b735944087827897f0ca9ffbff" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45821 } }
I20260926 07:23:32.415700 15299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:32.415771 15299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6666c0b735944087827897f0ca9ffbff, State: Initialized, Role: FOLLOWER
I20260926 07:23:32.416254 15299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c0b735944087827897f0ca9ffbff" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45821 } }
I20260926 07:23:32.416380 15299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:32.416520 15299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:32.416661 15299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:32.417722 15299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c0b735944087827897f0ca9ffbff" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45821 } }
I20260926 07:23:32.418139 15299 leader_election.cc:302] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6666c0b735944087827897f0ca9ffbff; no voters:
I20260926 07:23:32.418509 15299 leader_election.cc:288] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:32.418587 15304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:32.418991 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:32.403474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.28.235.254:45821)
0926 07:23:32.403904 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:32.403935 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:32.403978 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:23:32.404785 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:32.404807 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:32.405260 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:32.405616 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:32.406345 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:32.407820 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:32.408745 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:32.409398 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:32.409570 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:32.412438 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:32.412446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:32.412466 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:32.412511 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:32.415473 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:32.416049 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:32.416056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:32.416061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:32.416174 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:32.416486 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:32.416492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:32.416496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:32.417067 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:32.417288 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:32.417666 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 07:23:32.417996 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":109,"threads_started":1}
I20260926 07:23:32.418776 15304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [term 1 LEADER]: Becoming Leader. State: Replica: 6666c0b735944087827897f0ca9ffbff, State: Running, Role: LEADER
I20260926 07:23:32.419970 15299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:32.420207 15304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c0b735944087827897f0ca9ffbff" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45821 } }
I20260926 07:23:32.422041 15306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6666c0b735944087827897f0ca9ffbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c0b735944087827897f0ca9ffbff" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45821 } } }
I20260926 07:23:32.422180 15306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:32.422488 15305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6666c0b735944087827897f0ca9ffbff. Latest consensus state: current_term: 1 leader_uuid: "6666c0b735944087827897f0ca9ffbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c0b735944087827897f0ca9ffbff" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45821 } } }
I20260926 07:23:32.422962 15305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff [sys.catalog]: This master's current role is: LEADER
W20260926 07:23:32.424902 15318 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:23:32.424975 15318 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:23:32.425086 15319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:32.425848 15319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:32.431624 15319 catalog_manager.cc:1347] Generated new cluster ID: 5196404330c540a5919268780a47b4b2
I20260926 07:23:32.431742 15319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:32.441476 15319 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:32.442638 15319 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:32.455878 15319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff: Generated new TSK 0
I20260926 07:23:32.456717 15319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:23:32.488827 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45821
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:32.602867 15325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:32.603214 15325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:32.603295 15325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:32.607482 15325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:32.607591 15325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:32.607690 15325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:23:32.613238 15325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:32.614411 15325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:32.615475 15325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:32.625700 15331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:32.625780 15330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:32.625758 15333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:32.626083 15325 server_base.cc:1034] running on GCE node
I20260926 07:23:32.626511 15325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:32.627164 15325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:32.628399 15325 hybrid_clock.cc:648] HybridClock initialized: now 1790407412628371 us; error 40 us; skew 500 ppm
Sep 26 07:23:32 dist-test-slave-7zlw krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407412, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:32.631579 15325 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:23:32.632511 15325 webserver.cc:460] Webserver started at http://127.28.235.193:44895/ using document root <none> and password file <none>
I20260926 07:23:32.633111 15325 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:32.633178 15325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:32.633350 15325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:32.635365 15325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "7d858563097e4952a99e6633231be54e"
format_stamp: "Formatted at 2026-09-26 07:23:32 on dist-test-slave-7zlw"
server_key: "bb5a5e84eb686a84acfc3f2d3af1e9c2"
server_key_iv: "e56bdee2fe6a8c7be1f88ee81551c9d4"
server_key_version: "encryptionkey@0"
I20260926 07:23:32.635875 15325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "7d858563097e4952a99e6633231be54e"
format_stamp: "Formatted at 2026-09-26 07:23:32 on dist-test-slave-7zlw"
server_key: "bb5a5e84eb686a84acfc3f2d3af1e9c2"
server_key_iv: "e56bdee2fe6a8c7be1f88ee81551c9d4"
server_key_version: "encryptionkey@0"
I20260926 07:23:32.639845 15325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 07:23:32.642412 15340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:32.643873 15325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:23:32.644032 15325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "7d858563097e4952a99e6633231be54e"
format_stamp: "Formatted at 2026-09-26 07:23:32 on dist-test-slave-7zlw"
server_key: "bb5a5e84eb686a84acfc3f2d3af1e9c2"
server_key_iv: "e56bdee2fe6a8c7be1f88ee81551c9d4"
server_key_version: "encryptionkey@0"
I20260926 07:23:32.644155 15325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:32.671525 15325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:32.674971 15325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:32.675205 15325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:32.675853 15325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:32.676736 15325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:32.676781 15325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:32.676847 15325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:32.676878 15325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:32.686584 15325 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:45419
I20260926 07:23:32.686605 15453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:45419 every 8 connection(s)
I20260926 07:23:32.687558 15325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:23:32.696376 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 15325
I20260926 07:23:32.696547 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:23:32.696866 29615 external_mini_cluster.cc:1428] Setting key 917074aec14240ae86d6150710dbc3e8
Sep 26 07:23:32 dist-test-slave-7zlw krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407412, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:32.702627 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:32.689618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:39899 (local address 127.28.235.254:45821)
0926 07:23:32.689760 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:32.689763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:32.690651 (+ 888us) server_negotiation.cc:407] Connection header received
0926 07:23:32.692002 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:32.692006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:32.692067 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:32.692183 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:32.693578 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:32.694149 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:32.695355 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:32.695602 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:32.695741 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:32.698808 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:32.698812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:32.698815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:32.698847 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:32.700412 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:32.700964 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:32.700968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:32.700970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:32.701110 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:32.701475 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:32.701479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:32.701480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:32.701669 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:32.701780 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:32.702309 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 07:23:32.702444 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:23:32.703510 15456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:32.689936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:45821 (local address 127.28.235.193:39899)
0926 07:23:32.690474 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:32.690518 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:32.691777 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:32.692365 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:32.692372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:32.692762 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:32.693408 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:32.693425 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:32.694292 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:32.694296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:32.695205 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:32.695215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:32.695786 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:32.695791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:32.695965 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:32.696712 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:32.696730 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:32.698634 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:32.700554 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:32.700562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:32.700578 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:32.700852 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:32.701206 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:32.701209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:32.701211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:32.701359 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:32.701778 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:32.701785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:32.702062 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 07:23:32.702258 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 07:23:32.702617 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":110,"threads_started":1}
I20260926 07:23:32.704887 15454 heartbeater.cc:347] Connected to a master server at 127.28.235.254:45821
I20260926 07:23:32.705219 15454 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:32.705883 15454 heartbeater.cc:511] Master 127.28.235.254:45821 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:23:32.707769 15263 ts_manager.cc:194] Registered new tserver with Master: 7d858563097e4952a99e6633231be54e (127.28.235.193:45419)
I20260926 07:23:32.709023 15263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:39899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:23:32.729681 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45821
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:32.842514 15459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:32.842799 15459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:32.842844 15459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:32.847024 15459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:32.847095 15459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:32.847154 15459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:23:32.852723 15459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:32.854074 15459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:32.855114 15459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:32.864959 15464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:32.865060 15459 server_base.cc:1034] running on GCE node
W20260926 07:23:32.864996 15465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:32.864985 15467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:32.865660 15459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:32.866372 15459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:32.867573 15459 hybrid_clock.cc:648] HybridClock initialized: now 1790407412867559 us; error 25 us; skew 500 ppm
Sep 26 07:23:32 dist-test-slave-7zlw krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407412, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:32.870530 15459 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:23:32.871521 15459 webserver.cc:460] Webserver started at http://127.28.235.194:40841/ using document root <none> and password file <none>
I20260926 07:23:32.872115 15459 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:32.872179 15459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:32.872398 15459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:32.874541 15459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "dc49da71dc0e4365ae1d8a97d4959992"
format_stamp: "Formatted at 2026-09-26 07:23:32 on dist-test-slave-7zlw"
server_key: "fde8493ade497d0bfe4bbd5d51a3a3d7"
server_key_iv: "df7e6502b949be5c4f4331bce2abbc4b"
server_key_version: "encryptionkey@0"
I20260926 07:23:32.875108 15459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "dc49da71dc0e4365ae1d8a97d4959992"
format_stamp: "Formatted at 2026-09-26 07:23:32 on dist-test-slave-7zlw"
server_key: "fde8493ade497d0bfe4bbd5d51a3a3d7"
server_key_iv: "df7e6502b949be5c4f4331bce2abbc4b"
server_key_version: "encryptionkey@0"
I20260926 07:23:32.879033 15459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:23:32.881556 15474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:32.882851 15459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:23:32.882951 15459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "dc49da71dc0e4365ae1d8a97d4959992"
format_stamp: "Formatted at 2026-09-26 07:23:32 on dist-test-slave-7zlw"
server_key: "fde8493ade497d0bfe4bbd5d51a3a3d7"
server_key_iv: "df7e6502b949be5c4f4331bce2abbc4b"
server_key_version: "encryptionkey@0"
I20260926 07:23:32.883087 15459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:32.908910 15459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:32.912353 15459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:32.912601 15459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:32.913311 15459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:32.914178 15459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:32.914227 15459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:32.914268 15459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:32.914283 15459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:32.924065 15459 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:42849
I20260926 07:23:32.924090 15587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:42849 every 8 connection(s)
I20260926 07:23:32.924988 15459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:23:32.927042 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 15459
I20260926 07:23:32.927181 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:23:32.927507 29615 external_mini_cluster.cc:1428] Setting key d7c26310f4635721d46197777b8989fd
Sep 26 07:23:32 dist-test-slave-7zlw krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407412, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
I20260926 07:23:32.942123 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:32.927073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:43557 (local address 127.28.235.254:45821)
0926 07:23:32.927272 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:32.927277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:32.928181 (+ 904us) server_negotiation.cc:407] Connection header received
0926 07:23:32.930028 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:32.930061 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:32.930118 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:32.930191 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:32.931732 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:32.932503 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:32.934430 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:32.934669 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:32.934753 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:32.937910 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:32.937916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:32.937921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:32.937952 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:32.939587 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:32.940210 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:32.940213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:32.940214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:32.940357 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:32.940757 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:32.940759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:32.940761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:32.940951 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:32.941086 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:32.941840 (+ 754us) server_negotiation.cc:299] Negotiation successful
0926 07:23:32.941984 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:23:32.943290 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:32.927449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:45821 (local address 127.28.235.194:43557)
0926 07:23:32.928009 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:32.928061 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:32.929782 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:32.930416 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:32.930427 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:32.930975 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:32.931570 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:32.931584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:32.932650 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:32.932656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:32.934272 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:32.934285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:32.934851 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:32.934857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:32.935118 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:32.935832 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:32.935859 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:32.937692 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:32.939763 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:32.939772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:32.939800 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:32.940089 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:32.940449 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:32.940451 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:32.940452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:32.940656 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:32.941093 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:32.941100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:32.941517 (+ 417us) client_negotiation.cc:769] Sending connection context
0926 07:23:32.941715 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 07:23:32.942091 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":141,"threads_started":1}
I20260926 07:23:32.944748 15588 heartbeater.cc:347] Connected to a master server at 127.28.235.254:45821
I20260926 07:23:32.945003 15588 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:32.945601 15588 heartbeater.cc:511] Master 127.28.235.254:45821 requested a full tablet report, sending...
I20260926 07:23:32.946679 15263 ts_manager.cc:194] Registered new tserver with Master: dc49da71dc0e4365ae1d8a97d4959992 (127.28.235.194:42849)
I20260926 07:23:32.947285 15263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:43557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:23:32.964159 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45821
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:33.082490 15593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:33.082823 15593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:33.082870 15593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:33.087070 15593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:33.087165 15593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:33.087307 15593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:23:33.092728 15593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:45821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:33.093885 15593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:33.094892 15593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:23:33.104853 15593 server_base.cc:1034] running on GCE node
W20260926 07:23:33.104774 15601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:33.104774 15598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:33.104795 15599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:33.105463 15593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:33.106091 15593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:33.107311 15593 hybrid_clock.cc:648] HybridClock initialized: now 1790407413107299 us; error 31 us; skew 500 ppm
Sep 26 07:23:33 dist-test-slave-7zlw krb5kdc[15218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407413, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:33.110366 15593 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:23:33.111263 15593 webserver.cc:460] Webserver started at http://127.28.235.195:45203/ using document root <none> and password file <none>
I20260926 07:23:33.111824 15593 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:33.111884 15593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:33.112059 15593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:33.114341 15593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "30876e2e5deb4d778e98eb43ba3327b0"
format_stamp: "Formatted at 2026-09-26 07:23:33 on dist-test-slave-7zlw"
server_key: "341e41c5bb50fc460cae4f4029c844b5"
server_key_iv: "6966ae4918d0d40d67cec9430aee6b54"
server_key_version: "encryptionkey@0"
I20260926 07:23:33.114898 15593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "30876e2e5deb4d778e98eb43ba3327b0"
format_stamp: "Formatted at 2026-09-26 07:23:33 on dist-test-slave-7zlw"
server_key: "341e41c5bb50fc460cae4f4029c844b5"
server_key_iv: "6966ae4918d0d40d67cec9430aee6b54"
server_key_version: "encryptionkey@0"
I20260926 07:23:33.119421 15593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:23:33.122324 15608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:33.123795 15593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:23:33.123915 15593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "30876e2e5deb4d778e98eb43ba3327b0"
format_stamp: "Formatted at 2026-09-26 07:23:33 on dist-test-slave-7zlw"
server_key: "341e41c5bb50fc460cae4f4029c844b5"
server_key_iv: "6966ae4918d0d40d67cec9430aee6b54"
server_key_version: "encryptionkey@0"
I20260926 07:23:33.124014 15593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:33.149256 15593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:33.152364 15593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:33.152514 15593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:33.153167 15593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:33.153951 15593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:33.153997 15593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:33.154037 15593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:33.154052 15593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:33.163077 15593 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:42597
I20260926 07:23:33.163098 15721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:42597 every 8 connection(s)
I20260926 07:23:33.164078 15593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:23:33.171236 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 15593
I20260926 07:23:33.171386 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:23:33.171739 29615 external_mini_cluster.cc:1428] Setting key 1e346bef917ad66c2684656a03e26e9f
Sep 26 07:23:33 dist-test-slave-7zlw krb5kdc[15218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407413, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:33.179293 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.165971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:35755 (local address 127.28.235.254:45821)
0926 07:23:33.166125 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:33.166128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:33.167006 (+ 878us) server_negotiation.cc:407] Connection header received
0926 07:23:33.168472 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:33.168476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:33.168539 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:33.168654 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:33.170033 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.170691 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.172359 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.172667 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.172784 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.175524 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:33.175532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:33.175534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:33.175564 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:33.177110 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:33.177666 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:33.177670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:33.177672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:33.177755 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:33.178111 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:33.178116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:33.178117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:33.178321 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:33.178453 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:33.178996 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 07:23:33.179126 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:23:33.180189 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.166308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:45821 (local address 127.28.235.195:35755)
0926 07:23:33.166846 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:33.166890 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:33.168242 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:33.168839 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:33.168847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:33.169328 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:33.169840 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.169855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.170833 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.170837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.172221 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.172231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.172829 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.172834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.173089 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.173737 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:33.173759 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:33.175332 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:33.177231 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:33.177238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:33.177255 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:33.177553 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:33.177866 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:33.177869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:33.177871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:33.178014 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:33.178450 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:33.178457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:33.178751 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 07:23:33.178956 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 07:23:33.179306 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":130,"threads_started":1}
I20260926 07:23:33.181488 15722 heartbeater.cc:347] Connected to a master server at 127.28.235.254:45821
I20260926 07:23:33.181761 15722 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:33.182302 15722 heartbeater.cc:511] Master 127.28.235.254:45821 requested a full tablet report, sending...
I20260926 07:23:33.183468 15263 ts_manager.cc:194] Registered new tserver with Master: 30876e2e5deb4d778e98eb43ba3327b0 (127.28.235.195:42597)
I20260926 07:23:33.184064 15263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:35755
I20260926 07:23:33.186022 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:23:33.196081 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.188166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.28.235.254:45821)
0926 07:23:33.188322 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:33.188326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:33.188514 (+ 188us) server_negotiation.cc:407] Connection header received
0926 07:23:33.188669 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:33.188672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:33.188714 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:33.188793 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:33.189575 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.190115 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.190889 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.191104 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.191195 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.192773 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:33.192777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:33.192779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:33.192802 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:33.194134 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:33.194705 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:33.194707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:33.194709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:33.194806 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:33.195106 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:33.195108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:33.195109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:33.195372 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:33.195491 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:33.195750 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 07:23:33.195889 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:23:33.199013 15263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:23:33.202021 15263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:23:33.217690 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.211101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:45419 (local address 127.0.0.1:50504)
0926 07:23:33.211669 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:33.211684 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:33.211775 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:33.212712 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:33.212716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:33.212743 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:33.212858 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.212865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.213984 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.213989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.215735 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.215746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.216869 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.216873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.217122 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.217184 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 07:23:33.217322 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 07:23:33.217497 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":186,"thread_start_us":104,"threads_started":1}
I20260926 07:23:33.217690 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.211388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:42597 (local address 127.0.0.1:60532)
0926 07:23:33.212209 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:33.212221 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:33.212339 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:33.212846 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:33.212849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:33.212864 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:33.212968 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.212974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.214703 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.214709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.215607 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.215618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.216869 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.216873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.217127 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.217187 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 07:23:33.217313 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 07:23:33.217498 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":181,"thread_start_us":114,"threads_started":1}
I20260926 07:23:33.218510 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.210865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40652 (local address 127.28.235.194:42849)
0926 07:23:33.212016 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:33.212029 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:33.212052 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:23:33.212120 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:33.212125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:33.212338 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:33.212546 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:33.212886 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.214124 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.217051 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.217841 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.217982 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.218118 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:33.218235 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 07:23:33.218369 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":156,"threads_started":1}
I20260926 07:23:33.218510 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.211819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.28.235.195:42597)
0926 07:23:33.212148 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:33.212160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:33.212310 (+ 150us) server_negotiation.cc:407] Connection header received
0926 07:23:33.212518 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:33.212523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:33.212704 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:33.212856 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:33.213113 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.214335 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.215733 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.216640 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.216792 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.216938 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:33.217360 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 07:23:33.218353 (+ 993us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":139,"threads_started":1}
I20260926 07:23:33.218483 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.210673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:42849 (local address 127.0.0.1:40652)
0926 07:23:33.211239 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:33.211289 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:33.211424 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:33.212550 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:33.212556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:33.212578 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:33.212724 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.212731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.214268 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.214271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.215734 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.215746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.217969 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.217973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.218068 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.218117 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 07:23:33.218204 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:23:33.218313 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":138,"thread_start_us":96,"threads_started":1}
I20260926 07:23:33.219043 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.211560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.28.235.193:45419)
0926 07:23:33.211997 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:33.212005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:33.212032 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:23:33.212116 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:33.212121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:33.212333 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:33.212546 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:33.213040 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.213846 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.215876 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.216684 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.216809 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.216938 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:33.218788 (+ 1850us) server_negotiation.cc:299] Negotiation successful
0926 07:23:33.218898 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":142,"threads_started":1}
I20260926 07:23:33.222043 15522 tablet_service.cc:1467] Processing CreateTablet for tablet b98c968192e64330be11120100a8d41b (DEFAULT_TABLE table=test-table [id=ec8b5a4853ee442cbee930cf0c4fd5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:33.222043 15388 tablet_service.cc:1467] Processing CreateTablet for tablet b98c968192e64330be11120100a8d41b (DEFAULT_TABLE table=test-table [id=ec8b5a4853ee442cbee930cf0c4fd5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:33.223276 15388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98c968192e64330be11120100a8d41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:33.223276 15522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98c968192e64330be11120100a8d41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:33.227236 15656 tablet_service.cc:1467] Processing CreateTablet for tablet b98c968192e64330be11120100a8d41b (DEFAULT_TABLE table=test-table [id=ec8b5a4853ee442cbee930cf0c4fd5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:33.228371 15656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98c968192e64330be11120100a8d41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:33.231231 15739 tablet_bootstrap.cc:492] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e: Bootstrap starting.
I20260926 07:23:33.232179 15740 tablet_bootstrap.cc:492] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992: Bootstrap starting.
I20260926 07:23:33.234084 15739 tablet_bootstrap.cc:654] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:33.234227 15740 tablet_bootstrap.cc:654] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:33.234988 15739 log.cc:824] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:33.235251 15740 log.cc:824] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:33.235745 15742 tablet_bootstrap.cc:492] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: Bootstrap starting.
I20260926 07:23:33.236971 15739 tablet_bootstrap.cc:492] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e: No bootstrap required, opened a new log
I20260926 07:23:33.237270 15739 ts_tablet_manager.cc:1397] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260926 07:23:33.237314 15740 tablet_bootstrap.cc:492] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992: No bootstrap required, opened a new log
I20260926 07:23:33.237481 15740 ts_tablet_manager.cc:1397] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260926 07:23:33.238078 15742 tablet_bootstrap.cc:654] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:33.239398 15742 log.cc:824] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:33.241407 15740 raft_consensus.cc:348] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 } } peers { permanent_uuid: "30876e2e5deb4d778e98eb43ba3327b0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42597 } } peers { permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 } }
I20260926 07:23:33.241417 15739 raft_consensus.cc:348] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 } } peers { permanent_uuid: "30876e2e5deb4d778e98eb43ba3327b0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42597 } } peers { permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 } }
I20260926 07:23:33.241700 15739 raft_consensus.cc:374] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:33.241701 15740 raft_consensus.cc:374] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:33.241767 15740 raft_consensus.cc:729] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc49da71dc0e4365ae1d8a97d4959992, State: Initialized, Role: FOLLOWER
I20260926 07:23:33.241791 15739 raft_consensus.cc:729] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d858563097e4952a99e6633231be54e, State: Initialized, Role: FOLLOWER
I20260926 07:23:33.242156 15742 tablet_bootstrap.cc:492] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: No bootstrap required, opened a new log
I20260926 07:23:33.242271 15740 consensus_queue.cc:260] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 } } peers { permanent_uuid: "30876e2e5deb4d778e98eb43ba3327b0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42597 } } peers { permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 } }
I20260926 07:23:33.242372 15742 ts_tablet_manager.cc:1397] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:23:33.242298 15739 consensus_queue.cc:260] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 } } peers { permanent_uuid: "30876e2e5deb4d778e98eb43ba3327b0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42597 } } peers { permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 } }
I20260926 07:23:33.243180 15454 heartbeater.cc:503] Master 127.28.235.254:45821 was elected leader, sending a full tablet report...
I20260926 07:23:33.243206 15588 heartbeater.cc:503] Master 127.28.235.254:45821 was elected leader, sending a full tablet report...
I20260926 07:23:33.243353 15740 ts_tablet_manager.cc:1428] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260926 07:23:33.244642 15739 ts_tablet_manager.cc:1428] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260926 07:23:33.246603 15742 raft_consensus.cc:348] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 } } peers { permanent_uuid: "30876e2e5deb4d778e98eb43ba3327b0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42597 } } peers { permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 } }
I20260926 07:23:33.246944 15742 raft_consensus.cc:374] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:33.247020 15742 raft_consensus.cc:729] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30876e2e5deb4d778e98eb43ba3327b0, State: Initialized, Role: FOLLOWER
I20260926 07:23:33.247488 15742 consensus_queue.cc:260] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 } } peers { permanent_uuid: "30876e2e5deb4d778e98eb43ba3327b0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42597 } } peers { permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 } }
I20260926 07:23:33.248315 15722 heartbeater.cc:503] Master 127.28.235.254:45821 was elected leader, sending a full tablet report...
I20260926 07:23:33.248598 15742 ts_tablet_manager.cc:1428] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260926 07:23:33.415165 15723 tablet.cc:2369] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:23:33.426247 15589 tablet.cc:2369] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:23:33.437695 15747 raft_consensus.cc:482] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:33.437927 15747 raft_consensus.cc:504] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 } } peers { permanent_uuid: "30876e2e5deb4d778e98eb43ba3327b0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42597 } } peers { permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 } }
W20260926 07:23:33.439016 15455 tablet.cc:2369] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:23:33.439296 15747 leader_election.cc:288] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc49da71dc0e4365ae1d8a97d4959992 (127.28.235.194:42849), 7d858563097e4952a99e6633231be54e (127.28.235.193:45419)
I20260926 07:23:33.443583 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.439672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:42849 (local address 127.28.235.195:60087)
0926 07:23:33.439831 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:33.439847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:33.439990 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:33.440279 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:33.440282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:33.440303 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:33.440432 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.440443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.441438 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.441441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.442370 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.442380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.443069 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.443073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.443189 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.443210 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:23:33.443295 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:23:33.443398 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":27}
I20260926 07:23:33.443581 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.439877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:60087 (local address 127.28.235.194:42849)
0926 07:23:33.440038 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:33.440045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:33.440061 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:23:33.440118 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:33.440121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:33.440173 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:33.440265 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:33.440569 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.441320 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.442515 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.442949 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.443043 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.443084 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:33.443326 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 07:23:33.443416 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:23:33.443612 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.439672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:45419 (local address 127.28.235.195:36223)
0926 07:23:33.440328 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:33.440340 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:33.440432 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:33.440669 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:33.440672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:33.440689 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:33.440806 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.440813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.441648 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.441650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.442483 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:33.442494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.443169 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.443172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:33.443252 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.443318 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 07:23:33.443395 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:23:33.443457 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":269,"thread_start_us":151,"threads_started":1}
I20260926 07:23:33.443684 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.439778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:36223 (local address 127.28.235.193:45419)
0926 07:23:33.439930 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:33.439936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:33.440410 (+ 474us) server_negotiation.cc:407] Connection header received
0926 07:23:33.440515 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:33.440519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:33.440572 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:33.440641 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:33.440934 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.441534 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.442609 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.443054 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.443172 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.443220 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:33.443461 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 07:23:33.443556 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:23:33.444365 15542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98c968192e64330be11120100a8d41b" candidate_uuid: "30876e2e5deb4d778e98eb43ba3327b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc49da71dc0e4365ae1d8a97d4959992" is_pre_election: true
I20260926 07:23:33.444365 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98c968192e64330be11120100a8d41b" candidate_uuid: "30876e2e5deb4d778e98eb43ba3327b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d858563097e4952a99e6633231be54e" is_pre_election: true
I20260926 07:23:33.444686 15408 raft_consensus.cc:2454] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30876e2e5deb4d778e98eb43ba3327b0 in term 0.
I20260926 07:23:33.444686 15542 raft_consensus.cc:2454] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30876e2e5deb4d778e98eb43ba3327b0 in term 0.
I20260926 07:23:33.445217 15612 leader_election.cc:302] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30876e2e5deb4d778e98eb43ba3327b0, 7d858563097e4952a99e6633231be54e, dc49da71dc0e4365ae1d8a97d4959992; no voters:
I20260926 07:23:33.445519 15747 raft_consensus.cc:2781] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:23:33.445600 15747 raft_consensus.cc:482] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:23:33.445632 15747 raft_consensus.cc:3037] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:33.446841 15747 raft_consensus.cc:504] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 } } peers { permanent_uuid: "30876e2e5deb4d778e98eb43ba3327b0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42597 } } peers { permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 } }
I20260926 07:23:33.447193 15747 leader_election.cc:288] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [CANDIDATE]: Term 1 election: Requested vote from peers dc49da71dc0e4365ae1d8a97d4959992 (127.28.235.194:42849), 7d858563097e4952a99e6633231be54e (127.28.235.193:45419)
I20260926 07:23:33.447649 15542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98c968192e64330be11120100a8d41b" candidate_uuid: "30876e2e5deb4d778e98eb43ba3327b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc49da71dc0e4365ae1d8a97d4959992"
I20260926 07:23:33.447611 15408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98c968192e64330be11120100a8d41b" candidate_uuid: "30876e2e5deb4d778e98eb43ba3327b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d858563097e4952a99e6633231be54e"
I20260926 07:23:33.447762 15542 raft_consensus.cc:3037] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:33.447773 15408 raft_consensus.cc:3037] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:33.448979 15408 raft_consensus.cc:2454] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30876e2e5deb4d778e98eb43ba3327b0 in term 1.
I20260926 07:23:33.448988 15542 raft_consensus.cc:2454] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30876e2e5deb4d778e98eb43ba3327b0 in term 1.
I20260926 07:23:33.449344 15612 leader_election.cc:302] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30876e2e5deb4d778e98eb43ba3327b0, 7d858563097e4952a99e6633231be54e, dc49da71dc0e4365ae1d8a97d4959992; no voters:
I20260926 07:23:33.449493 15747 raft_consensus.cc:2781] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:33.449710 15747 raft_consensus.cc:686] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 1 LEADER]: Becoming Leader. State: Replica: 30876e2e5deb4d778e98eb43ba3327b0, State: Running, Role: LEADER
I20260926 07:23:33.450047 15747 consensus_queue.cc:237] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 } } peers { permanent_uuid: "30876e2e5deb4d778e98eb43ba3327b0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42597 } } peers { permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 } }
I20260926 07:23:33.453404 15262 catalog_manager.cc:5495] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30876e2e5deb4d778e98eb43ba3327b0 (127.28.235.195). New cstate: current_term: 1 leader_uuid: "30876e2e5deb4d778e98eb43ba3327b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30876e2e5deb4d778e98eb43ba3327b0" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 } health_report { overall_health: UNKNOWN } } }
I20260926 07:23:33.512639 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.508999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.28.235.195:42597)
0926 07:23:33.509167 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:33.509171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:33.509185 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:23:33.509253 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:33.509255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:33.509305 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:33.509394 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:23:33.509636 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.510140 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.511027 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.511228 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.511303 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.511552 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:23:33.512008 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:23:33.512120 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:33.512391 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 07:23:33.512458 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 07:23:33.518436 15408 raft_consensus.cc:1261] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e [term 1 FOLLOWER]: Refusing update from remote peer 30876e2e5deb4d778e98eb43ba3327b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:33.518446 15542 raft_consensus.cc:1261] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992 [term 1 FOLLOWER]: Refusing update from remote peer 30876e2e5deb4d778e98eb43ba3327b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:33.519174 15749 consensus_queue.cc:1035] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d858563097e4952a99e6633231be54e" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 45419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:33.519371 15747 consensus_queue.cc:1035] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc49da71dc0e4365ae1d8a97d4959992" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:33.527634 15752 mvcc.cc:205] Tried to move back new op lower bound from 7333508765764419584 to 7333508765493907456. Current Snapshot: MvccSnapshot[applied={T|T < 7333508765764419584}]
I20260926 07:23:33.530056 15753 mvcc.cc:205] Tried to move back new op lower bound from 7333508765764419584 to 7333508765493907456. Current Snapshot: MvccSnapshot[applied={T|T < 7333508765764419584}]
I20260926 07:23:33.537871 15261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:33.538086 15261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:33.540867 15261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6666c0b735944087827897f0ca9ffbff: Sending DeleteTablet for 3 replicas of tablet b98c968192e64330be11120100a8d41b
I20260926 07:23:33.541741 15522 tablet_service.cc:1514] Processing DeleteTablet for tablet b98c968192e64330be11120100a8d41b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:33 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:40652
I20260926 07:23:33.541780 15656 tablet_service.cc:1514] Processing DeleteTablet for tablet b98c968192e64330be11120100a8d41b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:33 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:60532
I20260926 07:23:33.541852 15388 tablet_service.cc:1514] Processing DeleteTablet for tablet b98c968192e64330be11120100a8d41b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:33 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:50504
I20260926 07:23:33.542353 15762 tablet_replica.cc:330] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e: stopping tablet replica
I20260926 07:23:33.542658 15764 tablet_replica.cc:330] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: stopping tablet replica
I20260926 07:23:33.542773 15762 raft_consensus.cc:2229] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:23:33.543021 15764 raft_consensus.cc:2229] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:23:33.543154 15762 raft_consensus.cc:2258] T b98c968192e64330be11120100a8d41b P 7d858563097e4952a99e6633231be54e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:33.543229 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 15325
I20260926 07:23:33.549353 15763 tablet_replica.cc:330] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992: stopping tablet replica
I20260926 07:23:33.549444 15764 raft_consensus.cc:2258] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:33.549780 15763 raft_consensus.cc:2229] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:23:33.550165 15763 raft_consensus.cc:2258] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 07:23:33.550901 15250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:45419 (error 108)
I20260926 07:23:33.551683 15764 ts_tablet_manager.cc:1905] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:33.551723 15763 ts_tablet_manager.cc:1905] T b98c968192e64330be11120100a8d41b P dc49da71dc0e4365ae1d8a97d4959992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:33.551844 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 15459
I20260926 07:23:33.554291 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.553886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:45419 (local address 127.0.0.1:57538)
0926 07:23:33.554105 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:33.554180 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:45419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260926 07:23:33.554478 15250 catalog_manager.cc:4584] TS 7d858563097e4952a99e6633231be54e (127.28.235.193:45419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b98c968192e64330be11120100a8d41b: Network error: Client connection negotiation failed: client connection to 127.28.235.193:45419: connect: Connection refused (error 111)
I20260926 07:23:33.556309 15764 ts_tablet_manager.cc:1918] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:33.556420 15764 log.cc:1196] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/b98c968192e64330be11120100a8d41b
I20260926 07:23:33.556795 15764 ts_tablet_manager.cc:1939] T b98c968192e64330be11120100a8d41b P 30876e2e5deb4d778e98eb43ba3327b0: Deleting consensus metadata
I20260926 07:23:33.558213 15249 catalog_manager.cc:4841] TS 30876e2e5deb4d778e98eb43ba3327b0 (127.28.235.195:42597): tablet b98c968192e64330be11120100a8d41b (table test-table [id=ec8b5a4853ee442cbee930cf0c4fd5ed]) successfully deleted
I20260926 07:23:33.558652 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 15593
I20260926 07:23:33.560545 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.560201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:42849 (local address 127.0.0.1:49614)
0926 07:23:33.560365 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:33.560436 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.194:42849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 07:23:33.564848 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 15231
2026-09-26T07:23:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2536 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T07:23:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:33Z Disabled control of system clock
I20260926 07:23:33.595918 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:32883
--webserver_interface=127.28.235.254
--webserver_port=0
--builtin_ntp_servers=127.28.235.212:40977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:23:33.706815 15769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:33.707083 15769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:33.707119 15769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:33.711088 15769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:33.711145 15769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:33.711160 15769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:33.711177 15769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:33.711192 15769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:33.715912 15769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:32883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:33.717067 15769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:33.718423 15769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:33.727312 15774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:33.727308 15777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:33.727308 15775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:33.727664 15769 server_base.cc:1034] running on GCE node
I20260926 07:23:33.728222 15769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:33.729457 15769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:33.730647 15769 hybrid_clock.cc:648] HybridClock initialized: now 1790407413730618 us; error 44 us; skew 500 ppm
I20260926 07:23:33.732514 15769 webserver.cc:460] Webserver started at http://127.28.235.254:33299/ using document root <none> and password file <none>
I20260926 07:23:33.733163 15769 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:33.733258 15769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:33.733487 15769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:33.735559 15769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "ff870cc876534610b6eefaf0735eb2f8"
format_stamp: "Formatted at 2026-09-26 07:23:33 on dist-test-slave-7zlw"
server_key: "3fceb437f3fef9e2af877dd4be421646"
server_key_iv: "a4061b69efc14e271f5cdeae3eff1bb3"
server_key_version: "encryptionkey@0"
I20260926 07:23:33.736140 15769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "ff870cc876534610b6eefaf0735eb2f8"
format_stamp: "Formatted at 2026-09-26 07:23:33 on dist-test-slave-7zlw"
server_key: "3fceb437f3fef9e2af877dd4be421646"
server_key_iv: "a4061b69efc14e271f5cdeae3eff1bb3"
server_key_version: "encryptionkey@0"
I20260926 07:23:33.740023 15769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:23:33.742688 15783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:33.744081 15769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:23:33.744307 15769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "ff870cc876534610b6eefaf0735eb2f8"
format_stamp: "Formatted at 2026-09-26 07:23:33 on dist-test-slave-7zlw"
server_key: "3fceb437f3fef9e2af877dd4be421646"
server_key_iv: "a4061b69efc14e271f5cdeae3eff1bb3"
server_key_version: "encryptionkey@0"
I20260926 07:23:33.744443 15769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:33.783334 15769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:33.784101 15769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:33.784312 15769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:33.791424 15769 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:32883
I20260926 07:23:33.791437 15835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:32883 every 8 connection(s)
I20260926 07:23:33.792367 15769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:33.795536 15836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:33.801198 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8: Bootstrap starting.
I20260926 07:23:33.802068 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 15769
I20260926 07:23:33.802204 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:33.802522 29615 external_mini_cluster.cc:1428] Setting key 15e49e1dd9d4d3c885ad57fe94683c6c
I20260926 07:23:33.804394 15836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:33.805264 15836 log.cc:824] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:33.807188 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8: No bootstrap required, opened a new log
I20260926 07:23:33.810827 15836 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff870cc876534610b6eefaf0735eb2f8" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 32883 } }
I20260926 07:23:33.811074 15836 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:33.811149 15836 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff870cc876534610b6eefaf0735eb2f8, State: Initialized, Role: FOLLOWER
I20260926 07:23:33.811600 15836 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff870cc876534610b6eefaf0735eb2f8" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 32883 } }
I20260926 07:23:33.811755 15836 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:33.811828 15836 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:33.811966 15836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:33.812093 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:33.803812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.28.235.254:32883)
0926 07:23:33.804579 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:33.804595 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:33.804631 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:23:33.805393 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:33.805423 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:33.805934 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:33.806299 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:33.806712 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.807716 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.808602 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:33.809172 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:33.809332 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:33.810247 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:33.810253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:33.810269 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:33.810645 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:33.810694 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:33.810713 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:33.810826 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:33.810976 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 07:23:33.811248 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":133,"threads_started":1}
I20260926 07:23:33.812893 15836 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff870cc876534610b6eefaf0735eb2f8" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 32883 } }
I20260926 07:23:33.813292 15836 leader_election.cc:302] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff870cc876534610b6eefaf0735eb2f8; no voters:
I20260926 07:23:33.813628 15836 leader_election.cc:288] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:33.813786 15841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:33.814047 15841 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [term 1 LEADER]: Becoming Leader. State: Replica: ff870cc876534610b6eefaf0735eb2f8, State: Running, Role: LEADER
I20260926 07:23:33.814447 15841 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff870cc876534610b6eefaf0735eb2f8" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 32883 } }
I20260926 07:23:33.814899 15836 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:33.815688 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff870cc876534610b6eefaf0735eb2f8. Latest consensus state: current_term: 1 leader_uuid: "ff870cc876534610b6eefaf0735eb2f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff870cc876534610b6eefaf0735eb2f8" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 32883 } } }
I20260926 07:23:33.815934 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:33.816115 15843 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff870cc876534610b6eefaf0735eb2f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff870cc876534610b6eefaf0735eb2f8" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 32883 } } }
I20260926 07:23:33.816241 15843 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:33.816349 15850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:33.819552 15850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:33.825263 15850 catalog_manager.cc:1347] Generated new cluster ID: 05de37eeca824dadb368899ea6e120aa
I20260926 07:23:33.825366 15850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:33.839607 15850 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:33.840922 15850 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:33.848889 15850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8: Generated new TSK 0
I20260926 07:23:33.849805 15850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:33.860246 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:32883
--builtin_ntp_servers=127.28.235.212:40977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:23:33.978093 15860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:33.978468 15860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:33.978509 15860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:33.982759 15860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:33.982821 15860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:33.982879 15860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:23:33.987958 15860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:32883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:33.989156 15860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:33.990587 15860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:34.000885 15868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:34.000869 15865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:34.000874 15866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:34.001134 15860 server_base.cc:1034] running on GCE node
I20260926 07:23:34.001725 15860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:34.002496 15860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:34.003683 15860 hybrid_clock.cc:648] HybridClock initialized: now 1790407414003671 us; error 36 us; skew 500 ppm
I20260926 07:23:34.005785 15860 webserver.cc:460] Webserver started at http://127.28.235.193:36485/ using document root <none> and password file <none>
I20260926 07:23:34.006479 15860 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:34.006573 15860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:34.006810 15860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:34.009197 15860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "8cba5cc1e9fc468e9c79b0fe853d8068"
format_stamp: "Formatted at 2026-09-26 07:23:34 on dist-test-slave-7zlw"
server_key: "1a2b5fc7c3a7a633c8d5db3847a41bcf"
server_key_iv: "1d762e2923438207cb28ba16784e8a74"
server_key_version: "encryptionkey@0"
I20260926 07:23:34.009824 15860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "8cba5cc1e9fc468e9c79b0fe853d8068"
format_stamp: "Formatted at 2026-09-26 07:23:34 on dist-test-slave-7zlw"
server_key: "1a2b5fc7c3a7a633c8d5db3847a41bcf"
server_key_iv: "1d762e2923438207cb28ba16784e8a74"
server_key_version: "encryptionkey@0"
I20260926 07:23:34.013797 15860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:23:34.016588 15874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:34.018049 15860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:23:34.018162 15860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "8cba5cc1e9fc468e9c79b0fe853d8068"
format_stamp: "Formatted at 2026-09-26 07:23:34 on dist-test-slave-7zlw"
server_key: "1a2b5fc7c3a7a633c8d5db3847a41bcf"
server_key_iv: "1d762e2923438207cb28ba16784e8a74"
server_key_version: "encryptionkey@0"
I20260926 07:23:34.018263 15860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:34.037137 15860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:34.037827 15860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:34.037982 15860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:34.038534 15860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:34.039474 15860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:34.039522 15860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:34.039562 15860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:34.039578 15860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:34.049424 15860 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:36569
I20260926 07:23:34.049456 15987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:36569 every 8 connection(s)
I20260926 07:23:34.050418 15860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:23:34.057471 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 15860
I20260926 07:23:34.057586 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:23:34.057879 29615 external_mini_cluster.cc:1428] Setting key 300175ede98d8c19e2fff1126d8e31e5
I20260926 07:23:34.060151 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:32883
--builtin_ntp_servers=127.28.235.212:40977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:23:34.061087 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.052364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:44909 (local address 127.28.235.254:32883)
0926 07:23:34.052522 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.052526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.053703 (+ 1177us) server_negotiation.cc:407] Connection header received
0926 07:23:34.055299 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.055303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.055378 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.055471 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:34.056477 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.057180 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.058773 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.059050 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.059178 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.060528 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:34.060531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:34.060533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:34.060545 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:34.060567 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:34.060575 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:34.060641 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.060846 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.060902 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:23:34.062287 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.052785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:32883 (local address 127.28.235.193:44909)
0926 07:23:34.053536 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.053580 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:34.055054 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:34.055669 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:34.055676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:34.055765 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:34.056318 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.056332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.057340 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.057349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.058638 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.058648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.059224 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.059228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.059486 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.060190 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:23:34.060193 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:34.060257 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:23:34.060360 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:34.060674 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:34.060687 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:34.060698 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 07:23:34.060845 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:23:34.061228 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":182,"threads_started":1}
I20260926 07:23:34.063644 15988 heartbeater.cc:347] Connected to a master server at 127.28.235.254:32883
I20260926 07:23:34.063897 15988 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:34.064502 15988 heartbeater.cc:511] Master 127.28.235.254:32883 requested a full tablet report, sending...
I20260926 07:23:34.066560 15795 ts_manager.cc:194] Registered new tserver with Master: 8cba5cc1e9fc468e9c79b0fe853d8068 (127.28.235.193:36569)
I20260926 07:23:34.068552 15795 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.193:44909
W20260926 07:23:34.177114 15991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:34.177397 15991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:34.177438 15991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:34.181478 15991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:34.181535 15991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:34.181589 15991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:23:34.186314 15991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:32883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:34.187726 15991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:34.189316 15991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:34.199342 15996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:34.199401 15999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:34.199535 15997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:34.199584 15991 server_base.cc:1034] running on GCE node
I20260926 07:23:34.200152 15991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:34.200793 15991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:34.202008 15991 hybrid_clock.cc:648] HybridClock initialized: now 1790407414201978 us; error 44 us; skew 500 ppm
I20260926 07:23:34.204012 15991 webserver.cc:460] Webserver started at http://127.28.235.194:43065/ using document root <none> and password file <none>
I20260926 07:23:34.204645 15991 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:34.204742 15991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:34.204960 15991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:34.207195 15991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "f37c720d0bd840aea4cc416559bb474f"
format_stamp: "Formatted at 2026-09-26 07:23:34 on dist-test-slave-7zlw"
server_key: "85226bfa0ad1cd74301077b40676e097"
server_key_iv: "687839b1af8c786e993cea0d99c02511"
server_key_version: "encryptionkey@0"
I20260926 07:23:34.207799 15991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "f37c720d0bd840aea4cc416559bb474f"
format_stamp: "Formatted at 2026-09-26 07:23:34 on dist-test-slave-7zlw"
server_key: "85226bfa0ad1cd74301077b40676e097"
server_key_iv: "687839b1af8c786e993cea0d99c02511"
server_key_version: "encryptionkey@0"
I20260926 07:23:34.211833 15991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:23:34.214453 16005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:34.215945 15991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:23:34.216073 15991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "f37c720d0bd840aea4cc416559bb474f"
format_stamp: "Formatted at 2026-09-26 07:23:34 on dist-test-slave-7zlw"
server_key: "85226bfa0ad1cd74301077b40676e097"
server_key_iv: "687839b1af8c786e993cea0d99c02511"
server_key_version: "encryptionkey@0"
I20260926 07:23:34.216202 15991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:34.236822 15991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:34.237666 15991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:34.237883 15991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:34.238543 15991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:34.239559 15991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:34.239634 15991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:34.239703 15991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:34.239758 15991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:34.249593 15991 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:42571
I20260926 07:23:34.249893 16118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:42571 every 8 connection(s)
I20260926 07:23:34.250618 15991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:23:34.258070 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 15991
I20260926 07:23:34.258212 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:23:34.258575 29615 external_mini_cluster.cc:1428] Setting key af0841d020fbe75e1a3a5d9e2c5ccabd
I20260926 07:23:34.260690 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.252534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:54119 (local address 127.28.235.254:32883)
0926 07:23:34.252686 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.252690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.253817 (+ 1127us) server_negotiation.cc:407] Connection header received
0926 07:23:34.255233 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.255237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.255300 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.255407 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:34.256306 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.256871 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.258443 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.258655 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.258793 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.260131 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:34.260136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:34.260139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:34.260154 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:34.260179 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:34.260192 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:34.260290 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.260488 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.260550 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:23:34.261065 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:32883
--builtin_ntp_servers=127.28.235.212:40977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:23:34.261756 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.252925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:32883 (local address 127.28.235.194:54119)
0926 07:23:34.253662 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.253707 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:34.255011 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:34.255588 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:34.255595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:34.255677 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:34.256162 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.256174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.257078 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.257081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.258302 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.258316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.258851 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.258856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.259070 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.259822 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:23:34.259824 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:34.259865 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:23:34.259937 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:34.260298 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:34.260306 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:34.260311 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:23:34.260471 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 07:23:34.260771 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":122,"threads_started":1}
I20260926 07:23:34.262876 16119 heartbeater.cc:347] Connected to a master server at 127.28.235.254:32883
I20260926 07:23:34.263130 16119 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:34.263669 16119 heartbeater.cc:511] Master 127.28.235.254:32883 requested a full tablet report, sending...
I20260926 07:23:34.264725 15795 ts_manager.cc:194] Registered new tserver with Master: f37c720d0bd840aea4cc416559bb474f (127.28.235.194:42571)
I20260926 07:23:34.265384 15795 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.194:54119
W20260926 07:23:34.379390 16122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:34.379700 16122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:34.379743 16122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:34.383978 16122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:34.384057 16122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:34.384117 16122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:23:34.389139 16122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:32883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:34.390254 16122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:34.391602 16122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:34.401292 16128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:34.401312 16130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:34.401312 16127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:34.401360 16122 server_base.cc:1034] running on GCE node
I20260926 07:23:34.401952 16122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:34.402657 16122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:34.403853 16122 hybrid_clock.cc:648] HybridClock initialized: now 1790407414403839 us; error 31 us; skew 500 ppm
I20260926 07:23:34.405876 16122 webserver.cc:460] Webserver started at http://127.28.235.195:43317/ using document root <none> and password file <none>
I20260926 07:23:34.406489 16122 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:34.406582 16122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:34.406814 16122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:34.408967 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "be8a0ad542634214a99ac54a77dba5a2"
format_stamp: "Formatted at 2026-09-26 07:23:34 on dist-test-slave-7zlw"
server_key: "5f52c3bd75eca5235d59470d371c4ad2"
server_key_iv: "761902713fc392847a813cc0bbe64420"
server_key_version: "encryptionkey@0"
I20260926 07:23:34.409607 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "be8a0ad542634214a99ac54a77dba5a2"
format_stamp: "Formatted at 2026-09-26 07:23:34 on dist-test-slave-7zlw"
server_key: "5f52c3bd75eca5235d59470d371c4ad2"
server_key_iv: "761902713fc392847a813cc0bbe64420"
server_key_version: "encryptionkey@0"
I20260926 07:23:34.413560 16122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:23:34.416477 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:34.418087 16122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:34.418239 16122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "be8a0ad542634214a99ac54a77dba5a2"
format_stamp: "Formatted at 2026-09-26 07:23:34 on dist-test-slave-7zlw"
server_key: "5f52c3bd75eca5235d59470d371c4ad2"
server_key_iv: "761902713fc392847a813cc0bbe64420"
server_key_version: "encryptionkey@0"
I20260926 07:23:34.418370 16122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:34.435338 16122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:34.436102 16122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:34.436318 16122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:34.436934 16122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:34.438025 16122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:34.438093 16122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:34.438165 16122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:34.438208 16122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:34.447556 16122 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:35461
I20260926 07:23:34.447595 16249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:35461 every 8 connection(s)
I20260926 07:23:34.448712 16122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:23:34.458419 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 16122
I20260926 07:23:34.458527 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.450829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:50279 (local address 127.28.235.254:32883)
0926 07:23:34.450981 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.450986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.452204 (+ 1218us) server_negotiation.cc:407] Connection header received
0926 07:23:34.453650 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.453653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.453702 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.453780 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:34.454702 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.455231 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.456425 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.456647 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.456742 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.457970 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:34.457973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:34.457975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:34.457990 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:34.458011 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:34.458020 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:34.458145 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.458257 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.458334 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:23:34.458556 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:23:34.458909 29615 external_mini_cluster.cc:1428] Setting key 7578e9975fc68f0977736d271d3660f8
I20260926 07:23:34.459443 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.451349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:32883 (local address 127.28.235.195:50279)
0926 07:23:34.452034 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.452075 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:34.453444 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:34.453969 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:34.453976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:34.454061 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:34.454560 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.454573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.455402 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.455405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.456293 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.456303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.456794 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.456797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.456961 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.457701 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:23:34.457703 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:34.457746 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:23:34.457813 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:34.458109 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:34.458116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:34.458120 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:23:34.458218 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:23:34.458451 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":126,"threads_started":1}
I20260926 07:23:34.460742 16250 heartbeater.cc:347] Connected to a master server at 127.28.235.254:32883
I20260926 07:23:34.460994 16250 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:34.461669 16250 heartbeater.cc:511] Master 127.28.235.254:32883 requested a full tablet report, sending...
I20260926 07:23:34.462726 15795 ts_manager.cc:194] Registered new tserver with Master: be8a0ad542634214a99ac54a77dba5a2 (127.28.235.195:35461)
I20260926 07:23:34.463318 15795 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.195:50279
I20260926 07:23:34.473465 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:23:34.479813 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.475580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.28.235.254:32883)
0926 07:23:34.475742 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.475747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.475896 (+ 149us) server_negotiation.cc:407] Connection header received
0926 07:23:34.476040 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.476043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.476096 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.476181 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:34.476479 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.477092 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.478003 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.478241 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.478351 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.479296 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:34.479300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:34.479303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:34.479315 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:34.479342 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:34.479350 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:34.479483 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.479608 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.479649 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:23:34.483307 15795 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:23:34.486480 15795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:23:34.503010 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.496141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:42571 (local address 127.0.0.1:41534)
0926 07:23:34.497140 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.497154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:34.497250 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:34.498162 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:34.498165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:34.498186 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:34.498279 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.498285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.499342 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.499347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.500952 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.500962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.502256 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.502260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.502515 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.502531 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:23:34.502643 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:23:34.502841 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":237,"thread_start_us":118,"threads_started":1}
I20260926 07:23:34.503032 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.496109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.28.235.195:35461)
0926 07:23:34.497364 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.497376 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.497567 (+ 191us) server_negotiation.cc:407] Connection header received
0926 07:23:34.497654 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.497661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.497853 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.498004 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:34.498272 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.499396 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.501247 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.502113 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.502261 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.502466 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.502696 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.502818 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":144,"threads_started":1}
I20260926 07:23:34.503636 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.496844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.28.235.194:42571)
0926 07:23:34.497564 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.497576 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.497600 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:23:34.497684 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.497687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.497887 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.498062 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:34.498410 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.499206 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.501132 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.502121 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.502266 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.502466 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.503321 (+ 855us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.503438 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":186,"threads_started":1}
I20260926 07:23:34.503698 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.495976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:35461 (local address 127.0.0.1:46980)
0926 07:23:34.497447 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.497460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:34.497557 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:34.498021 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:34.498024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:34.498042 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:34.498140 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.498149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.499536 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.499540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.501112 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.501123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.502256 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.502260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.502459 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.502514 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 07:23:34.502643 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 07:23:34.502841 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1397,"mutex_wait_us":777,"thread_start_us":113,"threads_started":1}
I20260926 07:23:34.504827 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.495976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:36569 (local address 127.0.0.1:54270)
0926 07:23:34.496882 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.496937 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:34.497105 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:34.497816 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:34.497822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:34.497844 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:34.498016 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.498025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.499536 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.499540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.501115 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.501123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.504422 (+ 3299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.504426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.504533 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.504559 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:23:34.504628 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:23:34.504704 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":146,"thread_start_us":95,"threads_started":1}
I20260926 07:23:34.505125 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.496237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54270 (local address 127.28.235.193:36569)
0926 07:23:34.496874 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.496892 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.497288 (+ 396us) server_negotiation.cc:407] Connection header received
0926 07:23:34.497381 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.497387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.497643 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.497809 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:34.498146 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.499396 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.502539 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.504271 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.504411 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.504595 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.504651 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.504762 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":138,"threads_started":1}
I20260926 07:23:34.506621 16184 tablet_service.cc:1467] Processing CreateTablet for tablet 0623150525394669b7473f1bcc5f0cfd (DEFAULT_TABLE table=test-table [id=7fecf82b154f47478bb0b46aa66bb89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:34.506623 16053 tablet_service.cc:1467] Processing CreateTablet for tablet 0623150525394669b7473f1bcc5f0cfd (DEFAULT_TABLE table=test-table [id=7fecf82b154f47478bb0b46aa66bb89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:34.507740 16184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0623150525394669b7473f1bcc5f0cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:34.507740 16053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0623150525394669b7473f1bcc5f0cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:34.510257 15922 tablet_service.cc:1467] Processing CreateTablet for tablet 0623150525394669b7473f1bcc5f0cfd (DEFAULT_TABLE table=test-table [id=7fecf82b154f47478bb0b46aa66bb89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:34.511526 15922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0623150525394669b7473f1bcc5f0cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:34.514305 16267 tablet_bootstrap.cc:492] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: Bootstrap starting.
I20260926 07:23:34.516127 16268 tablet_bootstrap.cc:492] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: Bootstrap starting.
I20260926 07:23:34.516899 16267 tablet_bootstrap.cc:654] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:34.517836 16267 log.cc:824] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:34.518688 16268 tablet_bootstrap.cc:654] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:34.519345 16270 tablet_bootstrap.cc:492] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068: Bootstrap starting.
I20260926 07:23:34.519572 16268 log.cc:824] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:34.520012 16267 tablet_bootstrap.cc:492] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: No bootstrap required, opened a new log
I20260926 07:23:34.520234 16267 ts_tablet_manager.cc:1397] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:23:34.521814 16268 tablet_bootstrap.cc:492] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: No bootstrap required, opened a new log
I20260926 07:23:34.521879 16270 tablet_bootstrap.cc:654] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:34.522058 16268 ts_tablet_manager.cc:1397] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260926 07:23:34.522754 16270 log.cc:824] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:34.524907 16267 raft_consensus.cc:348] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } }
I20260926 07:23:34.525208 16267 raft_consensus.cc:374] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:34.525269 16267 raft_consensus.cc:729] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8a0ad542634214a99ac54a77dba5a2, State: Initialized, Role: FOLLOWER
I20260926 07:23:34.525398 16270 tablet_bootstrap.cc:492] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068: No bootstrap required, opened a new log
I20260926 07:23:34.525569 16270 ts_tablet_manager.cc:1397] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:23:34.525796 16267 consensus_queue.cc:260] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } }
I20260926 07:23:34.526348 16268 raft_consensus.cc:348] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } }
I20260926 07:23:34.526628 16268 raft_consensus.cc:374] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:34.526724 16268 raft_consensus.cc:729] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37c720d0bd840aea4cc416559bb474f, State: Initialized, Role: FOLLOWER
I20260926 07:23:34.526842 16267 ts_tablet_manager.cc:1428] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:23:34.527215 16250 heartbeater.cc:503] Master 127.28.235.254:32883 was elected leader, sending a full tablet report...
I20260926 07:23:34.527199 16268 consensus_queue.cc:260] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } }
I20260926 07:23:34.527889 16119 heartbeater.cc:503] Master 127.28.235.254:32883 was elected leader, sending a full tablet report...
I20260926 07:23:34.528208 16268 ts_tablet_manager.cc:1428] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:23:34.529692 16270 raft_consensus.cc:348] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } }
I20260926 07:23:34.529930 16270 raft_consensus.cc:374] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:34.530009 16270 raft_consensus.cc:729] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cba5cc1e9fc468e9c79b0fe853d8068, State: Initialized, Role: FOLLOWER
I20260926 07:23:34.530441 16270 consensus_queue.cc:260] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } }
I20260926 07:23:34.531549 16270 ts_tablet_manager.cc:1428] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:23:34.531886 15988 heartbeater.cc:503] Master 127.28.235.254:32883 was elected leader, sending a full tablet report...
W20260926 07:23:34.551642 15989 tablet.cc:2369] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:23:34.605085 16273 raft_consensus.cc:482] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:34.605373 16273 raft_consensus.cc:504] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } }
I20260926 07:23:34.606719 16273 leader_election.cc:288] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cba5cc1e9fc468e9c79b0fe853d8068 (127.28.235.193:36569), f37c720d0bd840aea4cc416559bb474f (127.28.235.194:42571)
I20260926 07:23:34.609689 16275 raft_consensus.cc:482] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:34.609885 16275 raft_consensus.cc:504] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } }
I20260926 07:23:34.611392 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.606975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:42571 (local address 127.28.235.195:52229)
0926 07:23:34.607090 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.607108 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:34.607210 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:34.607522 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:34.607525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:34.607548 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:34.607685 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.607698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.608661 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.608664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.610038 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.610050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.610884 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.610888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.611000 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.611020 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:23:34.611119 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:23:34.611216 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 07:23:34.611897 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.607104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:52229 (local address 127.28.235.194:42571)
0926 07:23:34.607233 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.607239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.607257 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:23:34.607315 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.607318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.607376 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.607471 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:34.607828 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.608559 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.610190 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.610747 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.610894 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.610950 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.611673 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.611760 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:23:34.611961 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.607073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:49619 (local address 127.28.235.193:36569)
0926 07:23:34.607223 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.607227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.607752 (+ 525us) server_negotiation.cc:407] Connection header received
0926 07:23:34.607882 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.607886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.607936 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.608015 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:34.608265 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.608889 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.610495 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.611026 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.611144 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.611200 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.611737 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.611819 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:23:34.612317 16275 leader_election.cc:288] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be8a0ad542634214a99ac54a77dba5a2 (127.28.235.195:35461), f37c720d0bd840aea4cc416559bb474f (127.28.235.194:42571)
I20260926 07:23:34.612644 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.606980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:36569 (local address 127.28.235.195:49619)
0926 07:23:34.607647 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.607662 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:34.607776 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:34.608035 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:34.608037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:34.608052 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:34.608156 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.608161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.609044 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.609048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.610364 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.610373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.611281 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.611284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.611400 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.611436 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:23:34.612316 (+ 880us) client_negotiation.cc:240] Negotiation successful
0926 07:23:34.612456 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":303,"thread_start_us":127,"threads_started":1}
I20260926 07:23:34.612573 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0623150525394669b7473f1bcc5f0cfd" candidate_uuid: "be8a0ad542634214a99ac54a77dba5a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37c720d0bd840aea4cc416559bb474f" is_pre_election: true
I20260926 07:23:34.612919 16073 raft_consensus.cc:2454] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8a0ad542634214a99ac54a77dba5a2 in term 0.
I20260926 07:23:34.613154 15942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0623150525394669b7473f1bcc5f0cfd" candidate_uuid: "be8a0ad542634214a99ac54a77dba5a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" is_pre_election: true
I20260926 07:23:34.613355 15942 raft_consensus.cc:2454] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8a0ad542634214a99ac54a77dba5a2 in term 0.
I20260926 07:23:34.613497 16137 leader_election.cc:302] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be8a0ad542634214a99ac54a77dba5a2, f37c720d0bd840aea4cc416559bb474f; no voters:
I20260926 07:23:34.613997 16273 raft_consensus.cc:2781] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:23:34.614109 16273 raft_consensus.cc:482] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:23:34.614177 16273 raft_consensus.cc:3037] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:34.615478 16273 raft_consensus.cc:504] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } }
I20260926 07:23:34.615849 16273 leader_election.cc:288] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [CANDIDATE]: Term 1 election: Requested vote from peers 8cba5cc1e9fc468e9c79b0fe853d8068 (127.28.235.193:36569), f37c720d0bd840aea4cc416559bb474f (127.28.235.194:42571)
I20260926 07:23:34.616218 15942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0623150525394669b7473f1bcc5f0cfd" candidate_uuid: "be8a0ad542634214a99ac54a77dba5a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068"
I20260926 07:23:34.616252 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0623150525394669b7473f1bcc5f0cfd" candidate_uuid: "be8a0ad542634214a99ac54a77dba5a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37c720d0bd840aea4cc416559bb474f"
I20260926 07:23:34.616344 15942 raft_consensus.cc:3037] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:34.616389 16073 raft_consensus.cc:3037] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:34.617785 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.611799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:55615 (local address 127.28.235.194:42571)
0926 07:23:34.612110 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.612115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.613201 (+ 1086us) server_negotiation.cc:407] Connection header received
0926 07:23:34.613541 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.613544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.613594 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.613646 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:34.614132 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.614818 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.616446 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.617098 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.617224 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.617269 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.617540 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.617619 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":111,"threads_started":1}
I20260926 07:23:34.617791 15942 raft_consensus.cc:2454] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8a0ad542634214a99ac54a77dba5a2 in term 1.
I20260926 07:23:34.617835 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.611681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:33199 (local address 127.28.235.195:35461)
0926 07:23:34.612057 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.612063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.612078 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:23:34.612123 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.612127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.612181 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.612251 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:34.612667 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.613459 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.614974 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.615508 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.615621 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.615688 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.617579 (+ 1891us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.617665 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306}
I20260926 07:23:34.617830 16073 raft_consensus.cc:2454] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8a0ad542634214a99ac54a77dba5a2 in term 1.
I20260926 07:23:34.618073 16138 leader_election.cc:302] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cba5cc1e9fc468e9c79b0fe853d8068, be8a0ad542634214a99ac54a77dba5a2; no voters:
I20260926 07:23:34.618294 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.611376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:42571 (local address 127.28.235.193:55615)
0926 07:23:34.613113 (+ 1737us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.613129 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:34.613358 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:34.613838 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:34.613841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:34.613860 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:34.613994 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.614000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.614970 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.614974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.616296 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.616306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.617227 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.617231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.617333 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.617397 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 07:23:34.618052 (+ 655us) client_negotiation.cc:240] Negotiation successful
0926 07:23:34.618136 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":73,"thread_start_us":109,"threads_started":1}
I20260926 07:23:34.618578 16273 raft_consensus.cc:2781] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:34.618719 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.611343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:35461 (local address 127.28.235.193:33199)
0926 07:23:34.611833 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.611848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:34.611942 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:34.612362 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:34.612365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:34.612386 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:34.612489 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.612495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.613587 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.613590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.614794 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:34.614805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.615818 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.615824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:34.617403 (+ 1579us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.617431 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:23:34.618225 (+ 794us) client_negotiation.cc:240] Negotiation successful
0926 07:23:34.618317 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":1396,"thread_start_us":103,"threads_started":1}
I20260926 07:23:34.618754 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0623150525394669b7473f1bcc5f0cfd" candidate_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37c720d0bd840aea4cc416559bb474f" is_pre_election: true
I20260926 07:23:34.618821 16273 raft_consensus.cc:686] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 1 LEADER]: Becoming Leader. State: Replica: be8a0ad542634214a99ac54a77dba5a2, State: Running, Role: LEADER
I20260926 07:23:34.618942 16073 raft_consensus.cc:2379] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8cba5cc1e9fc468e9c79b0fe853d8068 in current term 1: Already voted for candidate be8a0ad542634214a99ac54a77dba5a2 in this term.
I20260926 07:23:34.619204 16273 consensus_queue.cc:237] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } }
I20260926 07:23:34.619261 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0623150525394669b7473f1bcc5f0cfd" candidate_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be8a0ad542634214a99ac54a77dba5a2" is_pre_election: true
I20260926 07:23:34.619903 15878 leader_election.cc:302] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8cba5cc1e9fc468e9c79b0fe853d8068; no voters: be8a0ad542634214a99ac54a77dba5a2, f37c720d0bd840aea4cc416559bb474f
I20260926 07:23:34.620257 16275 raft_consensus.cc:2726] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 07:23:34.623065 15795 catalog_manager.cc:5495] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to be8a0ad542634214a99ac54a77dba5a2 (127.28.235.195). New cstate: current_term: 1 leader_uuid: "be8a0ad542634214a99ac54a77dba5a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8a0ad542634214a99ac54a77dba5a2" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 35461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 } health_report { overall_health: UNKNOWN } } }
I20260926 07:23:34.659054 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.655060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.28.235.195:35461)
0926 07:23:34.655217 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:34.655221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:34.655236 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:23:34.655353 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:34.655355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:34.655411 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:34.655501 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:23:34.655823 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.656363 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.657375 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:34.657598 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:34.657685 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:34.658118 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:23:34.658565 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:23:34.658682 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:34.658805 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 07:23:34.658836 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:23:34.665192 16073 raft_consensus.cc:1261] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [term 1 FOLLOWER]: Refusing update from remote peer be8a0ad542634214a99ac54a77dba5a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:34.665201 15942 raft_consensus.cc:1261] T 0623150525394669b7473f1bcc5f0cfd P 8cba5cc1e9fc468e9c79b0fe853d8068 [term 1 FOLLOWER]: Refusing update from remote peer be8a0ad542634214a99ac54a77dba5a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:34.665895 16280 consensus_queue.cc:1035] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cba5cc1e9fc468e9c79b0fe853d8068" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:34.666144 16273 consensus_queue.cc:1035] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37c720d0bd840aea4cc416559bb474f" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:34.676383 16284 mvcc.cc:205] Tried to move back new op lower bound from 7333508770460250112 to 7333508770283458560. Current Snapshot: MvccSnapshot[applied={T|T < 7333508770460250112}]
I20260926 07:23:34.677284 16285 mvcc.cc:205] Tried to move back new op lower bound from 7333508770460250112 to 7333508770283458560. Current Snapshot: MvccSnapshot[applied={T|T < 7333508770460250112 or (T in {7333508770460250112})}]
I20260926 07:23:34.683292 15798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:34.683470 15798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:34.686203 15798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff870cc876534610b6eefaf0735eb2f8: Sending DeleteTablet for 3 replicas of tablet 0623150525394669b7473f1bcc5f0cfd
I20260926 07:23:34.687230 16184 tablet_service.cc:1514] Processing DeleteTablet for tablet 0623150525394669b7473f1bcc5f0cfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:34 UTC) from {username='slave'} at 127.0.0.1:46980
I20260926 07:23:34.687206 16053 tablet_service.cc:1514] Processing DeleteTablet for tablet 0623150525394669b7473f1bcc5f0cfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:34 UTC) from {username='slave'} at 127.0.0.1:41534
I20260926 07:23:34.687135 15922 tablet_service.cc:1514] Processing DeleteTablet for tablet 0623150525394669b7473f1bcc5f0cfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:34 UTC) from {username='slave'} at 127.0.0.1:54270
I20260926 07:23:34.687842 16293 tablet_replica.cc:330] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: stopping tablet replica
I20260926 07:23:34.688012 16295 tablet_replica.cc:330] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: stopping tablet replica
I20260926 07:23:34.688185 16293 raft_consensus.cc:2229] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:23:34.688324 16295 raft_consensus.cc:2229] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:23:34.688541 16293 raft_consensus.cc:2258] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:34.688599 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 15860
I20260926 07:23:34.688755 16295 raft_consensus.cc:2258] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:34.691872 16293 ts_tablet_manager.cc:1905] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:34.694979 16293 ts_tablet_manager.cc:1918] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:34.695096 16293 log.cc:1196] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/0623150525394669b7473f1bcc5f0cfd
I20260926 07:23:34.695442 16293 ts_tablet_manager.cc:1939] T 0623150525394669b7473f1bcc5f0cfd P f37c720d0bd840aea4cc416559bb474f: Deleting consensus metadata
I20260926 07:23:34.696849 16295 ts_tablet_manager.cc:1905] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:34.696892 15784 catalog_manager.cc:4841] TS f37c720d0bd840aea4cc416559bb474f (127.28.235.194:42571): tablet 0623150525394669b7473f1bcc5f0cfd (table test-table [id=7fecf82b154f47478bb0b46aa66bb89c]) successfully deleted
W20260926 07:23:34.697687 15785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:36569 (error 108)
I20260926 07:23:34.698552 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 15991
I20260926 07:23:34.698833 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:34.698481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:36569 (local address 127.0.0.1:54272)
0926 07:23:34.698664 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:34.698746 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:36569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 07:23:34.700716 16295 ts_tablet_manager.cc:1918] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:34.700874 16295 log.cc:1196] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/0623150525394669b7473f1bcc5f0cfd
W20260926 07:23:34.701145 15785 catalog_manager.cc:4584] TS 8cba5cc1e9fc468e9c79b0fe853d8068 (127.28.235.193:36569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0623150525394669b7473f1bcc5f0cfd: Network error: Client connection negotiation failed: client connection to 127.28.235.193:36569: connect: Connection refused (error 111)
I20260926 07:23:34.701234 16295 ts_tablet_manager.cc:1939] T 0623150525394669b7473f1bcc5f0cfd P be8a0ad542634214a99ac54a77dba5a2: Deleting consensus metadata
I20260926 07:23:34.702404 15787 catalog_manager.cc:4841] TS be8a0ad542634214a99ac54a77dba5a2 (127.28.235.195:35461): tablet 0623150525394669b7473f1bcc5f0cfd (table test-table [id=7fecf82b154f47478bb0b46aa66bb89c]) successfully deleted
I20260926 07:23:34.707907 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 16122
I20260926 07:23:34.714605 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 15769
2026-09-26T07:23:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1148 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:34 dist-test-slave-7zlw krb5kdc[16299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:34 dist-test-slave-7zlw krb5kdc[16299](info): set up 2 sockets
Sep 26 07:23:34 dist-test-slave-7zlw krb5kdc[16299](info): commencing operation
krb5kdc: starting...
W20260926 07:23:35.813365 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:35 dist-test-slave-7zlw krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:23:35.952760 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34603
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:34603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:36.074484 16312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:36.074770 16312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:36.074813 16312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:36.078987 16312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:36.079054 16312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:36.079075 16312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:36.079092 16312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:36.079108 16312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:36.084486 16312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:34603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:34603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:36.085673 16312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:36.086670 16312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:36.095510 16317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:36.095512 16320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:36.095558 16318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:36.095796 16312 server_base.cc:1034] running on GCE node
I20260926 07:23:36.096494 16312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:36.097827 16312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:36.099069 16312 hybrid_clock.cc:648] HybridClock initialized: now 1790407416099014 us; error 68 us; skew 500 ppm
Sep 26 07:23:36 dist-test-slave-7zlw krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407416, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:36.102154 16312 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:23:36.103083 16312 webserver.cc:460] Webserver started at http://127.28.235.254:41567/ using document root <none> and password file <none>
I20260926 07:23:36.103641 16312 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:36.103700 16312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:36.103873 16312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:36.106145 16312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "317419d56a5d4b59a50c21dd8016758f"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "8e3a946a6b67826492faaf7a44b008d6"
server_key_iv: "b7ca43409584051f2287fbb27dd2bf03"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.106709 16312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "317419d56a5d4b59a50c21dd8016758f"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "8e3a946a6b67826492faaf7a44b008d6"
server_key_iv: "b7ca43409584051f2287fbb27dd2bf03"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.110738 16312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:23:36.113281 16327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:36.114718 16312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:36.114830 16312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "317419d56a5d4b59a50c21dd8016758f"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "8e3a946a6b67826492faaf7a44b008d6"
server_key_iv: "b7ca43409584051f2287fbb27dd2bf03"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.114926 16312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:36.129582 16312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:36.132877 16312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:36.133075 16312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:36.140218 16312 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:34603
I20260926 07:23:36.140326 16379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:34603 every 8 connection(s)
I20260926 07:23:36.141284 16312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:36.144214 16380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:36.149885 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 16312
I20260926 07:23:36.149968 16380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f: Bootstrap starting.
I20260926 07:23:36.150066 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:36.150331 29615 external_mini_cluster.cc:1428] Setting key a410be40414da84eb8d085506e9a22fc
I20260926 07:23:36.152797 16380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:36.153726 16380 log.cc:824] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:36.156473 16380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f: No bootstrap required, opened a new log
Sep 26 07:23:36 dist-test-slave-7zlw krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:36.160921 16380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317419d56a5d4b59a50c21dd8016758f" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34603 } }
I20260926 07:23:36.161196 16380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:36.161316 16380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317419d56a5d4b59a50c21dd8016758f, State: Initialized, Role: FOLLOWER
I20260926 07:23:36.161835 16380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317419d56a5d4b59a50c21dd8016758f" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34603 } }
I20260926 07:23:36.162035 16380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:36.162104 16380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:36.162214 16380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:36.163496 16380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317419d56a5d4b59a50c21dd8016758f" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34603 } }
I20260926 07:23:36.163895 16380 leader_election.cc:302] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 317419d56a5d4b59a50c21dd8016758f; no voters:
I20260926 07:23:36.164291 16380 leader_election.cc:288] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:36.164628 16385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:36.164882 16385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [term 1 LEADER]: Becoming Leader. State: Replica: 317419d56a5d4b59a50c21dd8016758f, State: Running, Role: LEADER
I20260926 07:23:36.165329 16385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317419d56a5d4b59a50c21dd8016758f" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34603 } }
I20260926 07:23:36.165760 16380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:36.167035 16386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "317419d56a5d4b59a50c21dd8016758f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317419d56a5d4b59a50c21dd8016758f" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34603 } } }
I20260926 07:23:36.167109 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.151669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.28.235.254:34603)
0926 07:23:36.152185 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:36.152203 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:36.152245 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:23:36.153183 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:36.153208 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:36.153631 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:36.154066 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:36.155045 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.156623 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.157703 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.158423 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.158583 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.161411 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:36.161417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:36.161444 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:36.161480 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:36.163628 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:36.164174 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:36.164183 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:36.164193 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:36.164345 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:36.164697 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:36.164700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:36.164702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:36.165230 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:36.165430 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:36.165763 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 07:23:36.166075 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":139,"threads_started":1}
I20260926 07:23:36.167167 16386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:36.167517 16387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 317419d56a5d4b59a50c21dd8016758f. Latest consensus state: current_term: 1 leader_uuid: "317419d56a5d4b59a50c21dd8016758f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317419d56a5d4b59a50c21dd8016758f" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 34603 } } }
I20260926 07:23:36.167609 16387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f [sys.catalog]: This master's current role is: LEADER
W20260926 07:23:36.170919 16400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:23:36.171021 16400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:23:36.171113 16395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:36.171869 16395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:36.177237 16395 catalog_manager.cc:1347] Generated new cluster ID: 8a6e58cc659c40858f6d2ae4f3866c2b
I20260926 07:23:36.177317 16395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:36.184463 16395 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:36.185786 16395 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:36.190433 16395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f: Generated new TSK 0
I20260926 07:23:36.191277 16395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
Entry for principal kudu/127.28.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab.
I20260926 07:23:36.229660 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34603
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:36.356760 16406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:36.357126 16406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:36.357177 16406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:36.361333 16406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:36.361413 16406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:36.361475 16406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:23:36.367491 16406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.193.keytab
--principal=kudu/127.28.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:36.368670 16406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:36.369738 16406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:36.380162 16411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:36.380215 16412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:36.380162 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:36.380483 16406 server_base.cc:1034] running on GCE node
I20260926 07:23:36.380940 16406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:36.381693 16406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:36.382906 16406 hybrid_clock.cc:648] HybridClock initialized: now 1790407416382889 us; error 29 us; skew 500 ppm
Sep 26 07:23:36 dist-test-slave-7zlw krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407416, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:36.386217 16406 init.cc:377] Logged in from keytab as kudu/127.28.235.193@KRBTEST.COM (short username kudu)
I20260926 07:23:36.387202 16406 webserver.cc:460] Webserver started at http://127.28.235.193:45991/ using document root <none> and password file <none>
I20260926 07:23:36.387780 16406 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:36.387905 16406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:36.388154 16406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:36.390344 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "15dc3a8a14db4e69936b2052ea3af3aa"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "fae3fa6dabc51786f90e8c0c72c03861"
server_key_iv: "7df84c83b0cd7ec969d7fb552e1f6331"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.390920 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "15dc3a8a14db4e69936b2052ea3af3aa"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "fae3fa6dabc51786f90e8c0c72c03861"
server_key_iv: "7df84c83b0cd7ec969d7fb552e1f6331"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.394870 16406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 07:23:36.397473 16421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:36.398705 16406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:23:36.398866 16406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "15dc3a8a14db4e69936b2052ea3af3aa"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "fae3fa6dabc51786f90e8c0c72c03861"
server_key_iv: "7df84c83b0cd7ec969d7fb552e1f6331"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.398989 16406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:36.415855 16406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:36.419366 16406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:36.419582 16406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:36.420210 16406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:36.421164 16406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:36.421232 16406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:36.421300 16406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:36.421355 16406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:36.430913 16406 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:38427
I20260926 07:23:36.430943 16534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:38427 every 8 connection(s)
I20260926 07:23:36.431874 16406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:23:36.437561 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 16406
I20260926 07:23:36.437733 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:23:36.438037 29615 external_mini_cluster.cc:1428] Setting key d0c9d04781ef3dacd324a62658ea124b
Sep 26 07:23:36 dist-test-slave-7zlw krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407416, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.193@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:36.448083 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.433902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:39525 (local address 127.28.235.254:34603)
0926 07:23:36.434075 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:36.434079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:36.434939 (+ 860us) server_negotiation.cc:407] Connection header received
0926 07:23:36.436354 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:36.436357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:36.436415 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:36.436501 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:36.437800 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.438567 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.440020 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.440434 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.440567 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.443761 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:36.443768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:36.443774 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:36.443810 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:36.445754 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:36.446322 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:36.446328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:36.446332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:36.446435 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:36.446800 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:36.446804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:36.446806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:36.447078 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:36.447185 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:36.447717 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 07:23:36.447871 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:23:36.448927 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.434289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34603 (local address 127.28.235.193:39525)
0926 07:23:36.434768 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:36.434811 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:36.436082 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:36.436678 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:36.436685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:36.437111 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:36.437595 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.437608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.438731 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.438733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.439869 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.439881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.440626 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.440630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.440813 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.441583 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:36.441649 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:36.443572 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:36.445896 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:36.445901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:36.445915 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:36.446207 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:36.446544 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:36.446547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:36.446548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:36.446695 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:36.447183 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:36.447189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:36.447463 (+ 274us) client_negotiation.cc:769] Sending connection context
0926 07:23:36.447628 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 07:23:36.447925 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.28.235.194@KRBTEST.COM; defaulting to no policy
I20260926 07:23:36.450371 16535 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34603
I20260926 07:23:36.450634 16535 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:36.451238 16535 heartbeater.cc:511] Master 127.28.235.254:34603 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.194@KRBTEST.COM" created.
I20260926 07:23:36.453136 16344 ts_manager.cc:194] Registered new tserver with Master: 15dc3a8a14db4e69936b2052ea3af3aa (127.28.235.193:38427)
I20260926 07:23:36.454412 16344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.193@KRBTEST.COM'} at 127.28.235.193:39525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
Entry for principal kudu/127.28.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab.
I20260926 07:23:36.469529 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34603
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:36.582008 16540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:36.582324 16540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:36.582370 16540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:36.586576 16540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:36.586651 16540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:36.586715 16540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:23:36.592412 16540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.194.keytab
--principal=kudu/127.28.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:36.593663 16540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:36.594681 16540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:36.604748 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:36.604736 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:36.604732 16545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:36.605207 16540 server_base.cc:1034] running on GCE node
I20260926 07:23:36.605686 16540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:36.606411 16540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:36.607565 16540 hybrid_clock.cc:648] HybridClock initialized: now 1790407416607549 us; error 30 us; skew 500 ppm
Sep 26 07:23:36 dist-test-slave-7zlw krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407416, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:36.610982 16540 init.cc:377] Logged in from keytab as kudu/127.28.235.194@KRBTEST.COM (short username kudu)
I20260926 07:23:36.611968 16540 webserver.cc:460] Webserver started at http://127.28.235.194:35029/ using document root <none> and password file <none>
I20260926 07:23:36.612558 16540 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:36.612658 16540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:36.612887 16540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:36.615054 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "bf8832053f814804a9d6b34869b7ec43"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "14d131fd79226f659017a31b4ae77f2a"
server_key_iv: "63e8a005e79569bde1f5866b44cd4d6b"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.615696 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "bf8832053f814804a9d6b34869b7ec43"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "14d131fd79226f659017a31b4ae77f2a"
server_key_iv: "63e8a005e79569bde1f5866b44cd4d6b"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.619887 16540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:23:36.622519 16555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:36.623939 16540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:23:36.624086 16540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "bf8832053f814804a9d6b34869b7ec43"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "14d131fd79226f659017a31b4ae77f2a"
server_key_iv: "63e8a005e79569bde1f5866b44cd4d6b"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.624223 16540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:36.635800 16540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:36.639086 16540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:36.639250 16540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:36.639775 16540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:36.640547 16540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:36.640594 16540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:36.640631 16540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:36.640647 16540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:36.649595 16540 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:45429
I20260926 07:23:36.649616 16668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:45429 every 8 connection(s)
I20260926 07:23:36.650481 16540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:23:36.656427 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 16540
I20260926 07:23:36.656540 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:23:36.656822 29615 external_mini_cluster.cc:1428] Setting key 3efb1bd75308454fba3d893160cd5500
Sep 26 07:23:36 dist-test-slave-7zlw krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407416, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.194@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:36.666702 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.652575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:42511 (local address 127.28.235.254:34603)
0926 07:23:36.652743 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:36.652747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:36.653584 (+ 837us) server_negotiation.cc:407] Connection header received
0926 07:23:36.654994 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:36.654998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:36.655057 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:36.655136 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:36.656326 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.657049 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.658455 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.658663 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.658741 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.662008 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:36.662011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:36.662013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:36.662036 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:36.663739 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:36.664524 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:36.664527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:36.664528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:36.664606 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:36.665114 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:36.665117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:36.665118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:36.665378 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:36.665471 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:36.666275 (+ 804us) server_negotiation.cc:299] Negotiation successful
0926 07:23:36.666454 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:23:36.667874 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.652884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34603 (local address 127.28.235.194:42511)
0926 07:23:36.653439 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:36.653481 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:36.654774 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:36.655307 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:36.655314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:36.655656 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:36.656119 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.656159 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.657187 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.657192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.658319 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.658350 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.658826 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.658831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.659055 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.659726 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:36.659752 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:36.661832 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:36.663907 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:36.663915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:36.663938 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:36.664386 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:36.664706 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:36.664712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:36.664717 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:36.664936 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:36.665516 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:36.665525 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:36.665922 (+ 397us) client_negotiation.cc:769] Sending connection context
0926 07:23:36.666159 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 07:23:36.666602 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":179,"threads_started":1}
WARNING: no policy specified for kudu/127.28.235.195@KRBTEST.COM; defaulting to no policy
I20260926 07:23:36.669462 16669 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34603
I20260926 07:23:36.669703 16669 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:36.670312 16669 heartbeater.cc:511] Master 127.28.235.254:34603 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.195@KRBTEST.COM" created.
I20260926 07:23:36.671468 16344 ts_manager.cc:194] Registered new tserver with Master: bf8832053f814804a9d6b34869b7ec43 (127.28.235.194:45429)
I20260926 07:23:36.672320 16344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.194@KRBTEST.COM'} at 127.28.235.194:42511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
Entry for principal kudu/127.28.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab.
I20260926 07:23:36.689004 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34603
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:41865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:36.800936 16674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:36.801290 16674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:36.801390 16674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:36.805701 16674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:36.805814 16674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:36.805899 16674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:23:36.811304 16674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:41865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.195.keytab
--principal=kudu/127.28.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:34603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:36.812372 16674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:36.813474 16674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:36.823133 16679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:36.823184 16680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:36.823132 16682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:36.823444 16674 server_base.cc:1034] running on GCE node
I20260926 07:23:36.823949 16674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:36.824638 16674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:36.825845 16674 hybrid_clock.cc:648] HybridClock initialized: now 1790407416825801 us; error 66 us; skew 500 ppm
Sep 26 07:23:36 dist-test-slave-7zlw krb5kdc[16299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407416, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:36.828886 16674 init.cc:377] Logged in from keytab as kudu/127.28.235.195@KRBTEST.COM (short username kudu)
I20260926 07:23:36.829955 16674 webserver.cc:460] Webserver started at http://127.28.235.195:39311/ using document root <none> and password file <none>
I20260926 07:23:36.830552 16674 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:36.830648 16674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:36.830888 16674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:36.833060 16674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "72856d7a0c244d078bd077c9fe14c13d"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "48846e8f0cb9be28054916ac2b0aa6c6"
server_key_iv: "90516b1a3daf4b97b447b10dc0c403f1"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.833683 16674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "72856d7a0c244d078bd077c9fe14c13d"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "48846e8f0cb9be28054916ac2b0aa6c6"
server_key_iv: "90516b1a3daf4b97b447b10dc0c403f1"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.837885 16674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 07:23:36.840384 16689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:36.841718 16674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 07:23:36.841886 16674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "72856d7a0c244d078bd077c9fe14c13d"
format_stamp: "Formatted at 2026-09-26 07:23:36 on dist-test-slave-7zlw"
server_key: "48846e8f0cb9be28054916ac2b0aa6c6"
server_key_iv: "90516b1a3daf4b97b447b10dc0c403f1"
server_key_version: "encryptionkey@0"
I20260926 07:23:36.842011 16674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:36.872763 16674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:36.876302 16674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:36.876523 16674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:36.877241 16674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:36.878101 16674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:36.878173 16674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:36.878252 16674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:36.878296 16674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:36.887548 16674 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:41079
I20260926 07:23:36.887570 16802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:41079 every 8 connection(s)
I20260926 07:23:36.888489 16674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:23:36.896317 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 16674
I20260926 07:23:36.896481 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:23:36.896831 29615 external_mini_cluster.cc:1428] Setting key 62ae44a5269394022f633c8601208cec
Sep 26 07:23:36 dist-test-slave-7zlw krb5kdc[16299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407416, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.195@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:36.903072 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.890620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:48813 (local address 127.28.235.254:34603)
0926 07:23:36.890765 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:36.890768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:36.891585 (+ 817us) server_negotiation.cc:407] Connection header received
0926 07:23:36.892992 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:36.892996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:36.893083 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:36.893192 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:36.894368 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.894895 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.896062 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.896289 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.896403 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.899311 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:36.899315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:36.899318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:36.899352 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:36.900729 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:36.901395 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:36.901397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:36.901399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:36.901479 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:36.901849 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:36.901852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:36.901853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:36.902035 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:36.902115 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:36.902795 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 07:23:36.902935 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:23:36.903985 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.890951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:34603 (local address 127.28.235.195:48813)
0926 07:23:36.891424 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:36.891466 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:36.892753 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:36.893377 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:36.893384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:36.893736 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:36.894215 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.894228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.895023 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.895027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.895910 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.895919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.896447 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.896451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.896671 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.897393 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:23:36.897414 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:36.899145 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:36.900873 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:36.900880 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:36.900897 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:36.901275 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:36.901575 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:23:36.901577 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:23:36.901578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:23:36.901726 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:23:36.902155 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:36.902161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:36.902466 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 07:23:36.902689 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 07:23:36.903030 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":114,"threads_started":1}
I20260926 07:23:36.905202 16803 heartbeater.cc:347] Connected to a master server at 127.28.235.254:34603
I20260926 07:23:36.905478 16803 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:36.906034 16803 heartbeater.cc:511] Master 127.28.235.254:34603 requested a full tablet report, sending...
I20260926 07:23:36.907219 16344 ts_manager.cc:194] Registered new tserver with Master: 72856d7a0c244d078bd077c9fe14c13d (127.28.235.195:41079)
I20260926 07:23:36.907862 16344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.235.195@KRBTEST.COM'} at 127.28.235.195:48813
I20260926 07:23:36.911100 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:23:36.920934 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.913150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.28.235.254:34603)
0926 07:23:36.913289 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:36.913294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:36.913483 (+ 189us) server_negotiation.cc:407] Connection header received
0926 07:23:36.913551 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:36.913553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:36.913611 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:36.913692 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:36.914457 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.914985 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.915866 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.916093 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.916184 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.917792 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:36.917795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:36.917798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:36.917823 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:36.919238 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:36.919735 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:36.919737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:36.919738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:36.919814 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:36.920092 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:36.920094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:36.920095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:36.920362 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:36.920466 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:36.920712 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 07:23:36.920811 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:23:36.923967 16344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:23:36.927148 16344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:23:36.943045 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.937793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52376 (local address 127.28.235.194:45429)
0926 07:23:36.938164 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:36.938175 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:36.938230 (+ 55us) server_negotiation.cc:407] Connection header received
0926 07:23:36.938402 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:36.938408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:36.938576 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:36.938711 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:36.939033 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.939765 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.941704 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.942284 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.942428 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.942576 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:36.942782 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 07:23:36.942897 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":94,"threads_started":1}
I20260926 07:23:36.943042 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.936555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38427 (local address 127.0.0.1:38142)
0926 07:23:36.937428 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:36.937476 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:36.937609 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:36.938318 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:36.938323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:36.938346 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:36.938531 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.938538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.940006 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.940009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.941365 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.941374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.942518 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.942525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.942628 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.942649 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:23:36.942700 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 07:23:36.942885 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":127,"spinlock_wait_cycles":85248,"thread_start_us":86,"threads_started":1}
I20260926 07:23:36.943277 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.936609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.28.235.195:41079)
0926 07:23:36.937140 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:36.937154 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:36.937577 (+ 423us) server_negotiation.cc:407] Connection header received
0926 07:23:36.937806 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:36.937812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:36.938032 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:36.938167 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:36.938721 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.939873 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.941647 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.942489 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.942597 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.942721 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:36.943037 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 07:23:36.943139 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":125,"threads_started":1}
I20260926 07:23:36.943359 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.937451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.28.235.193:38427)
0926 07:23:36.937876 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:36.937893 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:36.937916 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:23:36.937988 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:36.937993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:36.938169 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:36.938333 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:36.938654 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.939873 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.941489 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.942368 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.942923 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.943069 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:36.943114 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 07:23:36.943198 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":84,"threads_started":1}
I20260926 07:23:36.943792 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.936435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:41079 (local address 127.0.0.1:50666)
0926 07:23:36.937426 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:36.937477 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:36.937652 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:36.938369 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:36.938371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:36.938393 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:36.938552 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.938561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.940051 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.940055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.941522 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.941533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.942791 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.942797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.942892 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.942911 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:23:36.943538 (+ 627us) client_negotiation.cc:240] Negotiation successful
0926 07:23:36.943639 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":162,"spinlock_wait_cycles":1664,"thread_start_us":111,"threads_started":1}
I20260926 07:23:36.944180 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:36.937302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:45429 (local address 127.0.0.1:52376)
0926 07:23:36.938146 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:36.938158 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:36.938249 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:36.938841 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:36.938843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:36.938857 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:36.938921 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.938926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.939901 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.939907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.941573 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:36.941582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:36.942415 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:36.942418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:36.942527 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:36.942607 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 07:23:36.942749 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 07:23:36.943747 (+ 998us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":157,"thread_start_us":90,"threads_started":1}
I20260926 07:23:36.947149 16603 tablet_service.cc:1467] Processing CreateTablet for tablet 24e8cc90920f45bc97868962e7cadfda (DEFAULT_TABLE table=test-table [id=a90c7082f1d84aa791c0fc1bd4d5c2a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:36.947022 16469 tablet_service.cc:1467] Processing CreateTablet for tablet 24e8cc90920f45bc97868962e7cadfda (DEFAULT_TABLE table=test-table [id=a90c7082f1d84aa791c0fc1bd4d5c2a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:36.947021 16737 tablet_service.cc:1467] Processing CreateTablet for tablet 24e8cc90920f45bc97868962e7cadfda (DEFAULT_TABLE table=test-table [id=a90c7082f1d84aa791c0fc1bd4d5c2a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:36.948038 16603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e8cc90920f45bc97868962e7cadfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:36.948067 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e8cc90920f45bc97868962e7cadfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:36.948066 16737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e8cc90920f45bc97868962e7cadfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:36.954473 16820 tablet_bootstrap.cc:492] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: Bootstrap starting.
I20260926 07:23:36.955687 16822 tablet_bootstrap.cc:492] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: Bootstrap starting.
I20260926 07:23:36.957072 16820 tablet_bootstrap.cc:654] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:36.957346 16821 tablet_bootstrap.cc:492] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa: Bootstrap starting.
I20260926 07:23:36.957487 16822 tablet_bootstrap.cc:654] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:36.958290 16820 log.cc:824] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:36.958738 16822 log.cc:824] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:36.959470 16821 tablet_bootstrap.cc:654] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:36.960201 16821 log.cc:824] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:36.960675 16820 tablet_bootstrap.cc:492] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: No bootstrap required, opened a new log
I20260926 07:23:36.960853 16822 tablet_bootstrap.cc:492] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: No bootstrap required, opened a new log
I20260926 07:23:36.960915 16820 ts_tablet_manager.cc:1397] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:23:36.961090 16822 ts_tablet_manager.cc:1397] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:23:36.962150 16821 tablet_bootstrap.cc:492] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa: No bootstrap required, opened a new log
I20260926 07:23:36.962328 16821 ts_tablet_manager.cc:1397] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 07:23:36.964118 16822 raft_consensus.cc:348] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 } } peers { permanent_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38427 } } peers { permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 } }
I20260926 07:23:36.964418 16822 raft_consensus.cc:374] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:36.964471 16822 raft_consensus.cc:729] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72856d7a0c244d078bd077c9fe14c13d, State: Initialized, Role: FOLLOWER
I20260926 07:23:36.964937 16822 consensus_queue.cc:260] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 } } peers { permanent_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38427 } } peers { permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 } }
I20260926 07:23:36.964972 16820 raft_consensus.cc:348] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 } } peers { permanent_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38427 } } peers { permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 } }
I20260926 07:23:36.965255 16820 raft_consensus.cc:374] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:36.965364 16820 raft_consensus.cc:729] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8832053f814804a9d6b34869b7ec43, State: Initialized, Role: FOLLOWER
I20260926 07:23:36.965814 16820 consensus_queue.cc:260] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 } } peers { permanent_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38427 } } peers { permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 } }
I20260926 07:23:36.966018 16822 ts_tablet_manager.cc:1428] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:23:36.966817 16803 heartbeater.cc:503] Master 127.28.235.254:34603 was elected leader, sending a full tablet report...
I20260926 07:23:36.966892 16820 ts_tablet_manager.cc:1428] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:23:36.966825 16821 raft_consensus.cc:348] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 } } peers { permanent_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38427 } } peers { permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 } }
I20260926 07:23:36.967092 16821 raft_consensus.cc:374] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:36.967149 16821 raft_consensus.cc:729] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15dc3a8a14db4e69936b2052ea3af3aa, State: Initialized, Role: FOLLOWER
I20260926 07:23:36.967206 16669 heartbeater.cc:503] Master 127.28.235.254:34603 was elected leader, sending a full tablet report...
I20260926 07:23:36.967536 16821 consensus_queue.cc:260] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 } } peers { permanent_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38427 } } peers { permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 } }
I20260926 07:23:36.968487 16821 ts_tablet_manager.cc:1428] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:23:36.968843 16535 heartbeater.cc:503] Master 127.28.235.254:34603 was elected leader, sending a full tablet report...
I20260926 07:23:37.015196 16828 raft_consensus.cc:482] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:37.015442 16828 raft_consensus.cc:504] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 } } peers { permanent_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38427 } } peers { permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 } }
I20260926 07:23:37.016592 16828 leader_election.cc:288] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72856d7a0c244d078bd077c9fe14c13d (127.28.235.195:41079), bf8832053f814804a9d6b34869b7ec43 (127.28.235.194:45429)
I20260926 07:23:37.021281 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.016939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:56753 (local address 127.28.235.194:45429)
0926 07:23:37.017091 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:37.017097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:37.017420 (+ 323us) server_negotiation.cc:407] Connection header received
0926 07:23:37.017582 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:37.017587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:37.017652 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:37.017722 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:37.018133 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.018616 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.020255 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.020715 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.020823 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.020865 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:37.021075 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 07:23:37.021151 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:23:37.021412 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.016866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:45429 (local address 127.28.235.193:56753)
0926 07:23:37.017286 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:37.017298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:37.017459 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:37.017754 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:37.017759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:37.017806 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:37.017995 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:37.018001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.018734 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.018740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:37.020129 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:37.020140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.020838 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.020841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:37.020918 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.020940 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:23:37.021073 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 07:23:37.021151 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":82,"spinlock_wait_cycles":8704,"thread_start_us":121,"threads_started":1}
I20260926 07:23:37.021955 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.016866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:41079 (local address 127.28.235.193:37865)
0926 07:23:37.017272 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:37.017285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:37.017466 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:37.017805 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:37.017808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:37.017824 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:37.017974 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:37.017979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.018898 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.018901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:37.020129 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:37.020140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.021551 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.021557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:37.021674 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.021697 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:23:37.021774 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:23:37.021852 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":47,"spinlock_wait_cycles":50432,"thread_start_us":157,"threads_started":1}
I20260926 07:23:37.022069 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e8cc90920f45bc97868962e7cadfda" candidate_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8832053f814804a9d6b34869b7ec43" is_pre_election: true
I20260926 07:23:37.022400 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.016942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:37865 (local address 127.28.235.195:41079)
0926 07:23:37.017091 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:37.017097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:37.017380 (+ 283us) server_negotiation.cc:407] Connection header received
0926 07:23:37.017635 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:37.017637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:37.017685 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:37.017794 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:37.018087 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.018789 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.020914 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.021448 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.021552 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.021617 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:37.022209 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 07:23:37.022292 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:23:37.022416 16623 raft_consensus.cc:2454] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15dc3a8a14db4e69936b2052ea3af3aa in term 0.
I20260926 07:23:37.022920 16423 leader_election.cc:302] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15dc3a8a14db4e69936b2052ea3af3aa, bf8832053f814804a9d6b34869b7ec43; no voters:
I20260926 07:23:37.023015 16757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e8cc90920f45bc97868962e7cadfda" candidate_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72856d7a0c244d078bd077c9fe14c13d" is_pre_election: true
I20260926 07:23:37.023262 16757 raft_consensus.cc:2454] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15dc3a8a14db4e69936b2052ea3af3aa in term 0.
I20260926 07:23:37.023272 16828 raft_consensus.cc:2781] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:23:37.023365 16828 raft_consensus.cc:482] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:23:37.023447 16828 raft_consensus.cc:3037] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:37.024554 16828 raft_consensus.cc:504] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 } } peers { permanent_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38427 } } peers { permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 } }
I20260926 07:23:37.024864 16828 leader_election.cc:288] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [CANDIDATE]: Term 1 election: Requested vote from peers 72856d7a0c244d078bd077c9fe14c13d (127.28.235.195:41079), bf8832053f814804a9d6b34869b7ec43 (127.28.235.194:45429)
I20260926 07:23:37.025246 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e8cc90920f45bc97868962e7cadfda" candidate_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8832053f814804a9d6b34869b7ec43"
I20260926 07:23:37.025319 16757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e8cc90920f45bc97868962e7cadfda" candidate_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72856d7a0c244d078bd077c9fe14c13d"
I20260926 07:23:37.025411 16623 raft_consensus.cc:3037] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:37.025429 16757 raft_consensus.cc:3037] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:37.026741 16757 raft_consensus.cc:2454] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15dc3a8a14db4e69936b2052ea3af3aa in term 1.
I20260926 07:23:37.026746 16623 raft_consensus.cc:2454] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15dc3a8a14db4e69936b2052ea3af3aa in term 1.
I20260926 07:23:37.027073 16423 leader_election.cc:302] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 15dc3a8a14db4e69936b2052ea3af3aa, 72856d7a0c244d078bd077c9fe14c13d, bf8832053f814804a9d6b34869b7ec43; no voters:
I20260926 07:23:37.027227 16828 raft_consensus.cc:2781] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:37.027433 16828 raft_consensus.cc:686] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [term 1 LEADER]: Becoming Leader. State: Replica: 15dc3a8a14db4e69936b2052ea3af3aa, State: Running, Role: LEADER
I20260926 07:23:37.027838 16828 consensus_queue.cc:237] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 } } peers { permanent_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38427 } } peers { permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 } }
I20260926 07:23:37.031306 16343 catalog_manager.cc:5495] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 15dc3a8a14db4e69936b2052ea3af3aa (127.28.235.193). New cstate: current_term: 1 leader_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15dc3a8a14db4e69936b2052ea3af3aa" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 } health_report { overall_health: UNKNOWN } } }
I20260926 07:23:37.042742 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.039062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.28.235.193:38427)
0926 07:23:37.039213 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:37.039218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:37.039233 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:23:37.039329 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:37.039332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:37.039387 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:37.039505 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:23:37.039766 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.040298 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.041233 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.041456 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.041567 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.041877 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:23:37.042315 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:23:37.042457 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:37.042522 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 07:23:37.042592 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:23:37.049211 16623 raft_consensus.cc:1261] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43 [term 1 FOLLOWER]: Refusing update from remote peer 15dc3a8a14db4e69936b2052ea3af3aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:37.049217 16757 raft_consensus.cc:1261] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d [term 1 FOLLOWER]: Refusing update from remote peer 15dc3a8a14db4e69936b2052ea3af3aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:37.049914 16828 consensus_queue.cc:1035] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "72856d7a0c244d078bd077c9fe14c13d" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:37.050129 16831 consensus_queue.cc:1035] T 24e8cc90920f45bc97868962e7cadfda P 15dc3a8a14db4e69936b2052ea3af3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8832053f814804a9d6b34869b7ec43" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 45429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:37.067270 16343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:37.067462 16343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:37.070282 16343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 317419d56a5d4b59a50c21dd8016758f: Sending DeleteTablet for 3 replicas of tablet 24e8cc90920f45bc97868962e7cadfda
I20260926 07:23:37.071215 16737 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e8cc90920f45bc97868962e7cadfda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:37 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:50666
I20260926 07:23:37.071270 16603 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e8cc90920f45bc97868962e7cadfda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:37 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:52376
I20260926 07:23:37.071470 16469 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e8cc90920f45bc97868962e7cadfda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:37 UTC) from {username='kudu', principal='kudu/127.28.235.254@KRBTEST.COM'} at 127.0.0.1:38142
I20260926 07:23:37.071967 16844 tablet_replica.cc:330] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: stopping tablet replica
I20260926 07:23:37.072268 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 16406
I20260926 07:23:37.073556 16845 tablet_replica.cc:330] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: stopping tablet replica
I20260926 07:23:37.073863 16845 raft_consensus.cc:2229] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:23:37.073871 16844 raft_consensus.cc:2229] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:23:37.074189 16844 raft_consensus.cc:2258] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:37.074195 16845 raft_consensus.cc:2258] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:37.075707 16844 ts_tablet_manager.cc:1905] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:37.075708 16845 ts_tablet_manager.cc:1905] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:37.078478 16845 ts_tablet_manager.cc:1918] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:37.078478 16844 ts_tablet_manager.cc:1918] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:37.078616 16845 log.cc:1196] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/24e8cc90920f45bc97868962e7cadfda
I20260926 07:23:37.078616 16844 log.cc:1196] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/24e8cc90920f45bc97868962e7cadfda
I20260926 07:23:37.078970 16845 ts_tablet_manager.cc:1939] T 24e8cc90920f45bc97868962e7cadfda P 72856d7a0c244d078bd077c9fe14c13d: Deleting consensus metadata
I20260926 07:23:37.078970 16844 ts_tablet_manager.cc:1939] T 24e8cc90920f45bc97868962e7cadfda P bf8832053f814804a9d6b34869b7ec43: Deleting consensus metadata
W20260926 07:23:37.079651 16329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:38427 (error 108)
I20260926 07:23:37.079895 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 16540
I20260926 07:23:37.080840 16328 catalog_manager.cc:4841] TS 72856d7a0c244d078bd077c9fe14c13d (127.28.235.195:41079): tablet 24e8cc90920f45bc97868962e7cadfda (table test-table [id=a90c7082f1d84aa791c0fc1bd4d5c2a6]) successfully deleted
I20260926 07:23:37.081674 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.081324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:38427 (local address 127.0.0.1:38150)
0926 07:23:37.081486 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:37.081567 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:38427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:23:37.081898 16329 catalog_manager.cc:4584] TS 15dc3a8a14db4e69936b2052ea3af3aa (127.28.235.193:38427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24e8cc90920f45bc97868962e7cadfda: Network error: Client connection negotiation failed: client connection to 127.28.235.193:38427: connect: Connection refused (error 111)
I20260926 07:23:37.086239 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 16674
I20260926 07:23:37.087107 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.086413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:45429 (local address 127.0.0.1:52388)
0926 07:23:37.086938 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:37.087003 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.194:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
I20260926 07:23:37.091876 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 16312
2026-09-26T07:23:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2377 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T07:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:37Z Disabled control of system clock
I20260926 07:23:37.122845 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:39573
--webserver_interface=127.28.235.254
--webserver_port=0
--builtin_ntp_servers=127.28.235.212:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:39573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:23:37.239554 16851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:37.239856 16851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:37.239892 16851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:37.244123 16851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:37.244202 16851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:37.244217 16851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:37.244235 16851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:37.244251 16851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:37.249326 16851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:39573
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:39573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:37.250522 16851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:37.251965 16851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:37.261197 16856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:37.261283 16859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:37.261216 16857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:37.261345 16851 server_base.cc:1034] running on GCE node
I20260926 07:23:37.262188 16851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:37.263378 16851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:37.264628 16851 hybrid_clock.cc:648] HybridClock initialized: now 1790407417264603 us; error 34 us; skew 500 ppm
I20260926 07:23:37.266768 16851 webserver.cc:460] Webserver started at http://127.28.235.254:38859/ using document root <none> and password file <none>
I20260926 07:23:37.267398 16851 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:37.267494 16851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:37.267726 16851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:37.270004 16851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "b9a8b70463914e0ebba376a2cf9609ba"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "08db7548b41eda816d12916af5adc656"
server_key_iv: "adf840f41ba0cfb62a9d457a21cc0b6e"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.270592 16851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "b9a8b70463914e0ebba376a2cf9609ba"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "08db7548b41eda816d12916af5adc656"
server_key_iv: "adf840f41ba0cfb62a9d457a21cc0b6e"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.274494 16851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:23:37.277168 16865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:37.278609 16851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:23:37.278788 16851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "b9a8b70463914e0ebba376a2cf9609ba"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "08db7548b41eda816d12916af5adc656"
server_key_iv: "adf840f41ba0cfb62a9d457a21cc0b6e"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.278927 16851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:37.301786 16851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:37.302636 16851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:37.302870 16851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:37.310364 16851 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:39573
I20260926 07:23:37.310416 16917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:39573 every 8 connection(s)
I20260926 07:23:37.311383 16851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:37.314842 16918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:37.319484 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 16851
I20260926 07:23:37.319628 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:37.319904 29615 external_mini_cluster.cc:1428] Setting key 22f15f629e34f0ab4738bb40df87ec7c
I20260926 07:23:37.321420 16918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba: Bootstrap starting.
I20260926 07:23:37.323814 16918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:37.324620 16918 log.cc:824] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:37.326646 16918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba: No bootstrap required, opened a new log
I20260926 07:23:37.330224 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.321526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.28.235.254:39573)
0926 07:23:37.322282 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:37.322302 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:37.322341 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:23:37.323034 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:37.323051 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:37.323474 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:37.323891 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:37.324361 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.325498 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.326468 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.327052 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.327238 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.328343 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:37.328348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:37.328363 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:37.328811 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:37.328871 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:37.328892 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:37.329022 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:37.329184 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 07:23:37.329450 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":155,"threads_started":1}
I20260926 07:23:37.330945 16918 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8b70463914e0ebba376a2cf9609ba" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39573 } }
I20260926 07:23:37.331228 16918 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:37.331282 16918 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a8b70463914e0ebba376a2cf9609ba, State: Initialized, Role: FOLLOWER
I20260926 07:23:37.331753 16918 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8b70463914e0ebba376a2cf9609ba" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39573 } }
I20260926 07:23:37.331879 16918 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:37.331960 16918 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:37.332083 16918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:37.333293 16918 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8b70463914e0ebba376a2cf9609ba" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39573 } }
I20260926 07:23:37.333644 16918 leader_election.cc:302] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9a8b70463914e0ebba376a2cf9609ba; no voters:
I20260926 07:23:37.333956 16918 leader_election.cc:288] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:37.334115 16923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:37.334448 16923 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [term 1 LEADER]: Becoming Leader. State: Replica: b9a8b70463914e0ebba376a2cf9609ba, State: Running, Role: LEADER
I20260926 07:23:37.334755 16923 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8b70463914e0ebba376a2cf9609ba" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39573 } }
I20260926 07:23:37.335299 16918 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:37.336449 16925 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9a8b70463914e0ebba376a2cf9609ba. Latest consensus state: current_term: 1 leader_uuid: "b9a8b70463914e0ebba376a2cf9609ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8b70463914e0ebba376a2cf9609ba" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39573 } } }
I20260926 07:23:37.336586 16925 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:37.336959 16932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:37.338037 16924 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9a8b70463914e0ebba376a2cf9609ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a8b70463914e0ebba376a2cf9609ba" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 39573 } } }
I20260926 07:23:37.338162 16924 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:37.340390 16932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:37.346148 16932 catalog_manager.cc:1347] Generated new cluster ID: f3a7bb4cd41842c7b8fd5306d2256722
I20260926 07:23:37.346237 16932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:37.358409 16932 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:37.359393 16932 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:37.370240 16932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba: Generated new TSK 0
I20260926 07:23:37.371011 16932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:37.376029 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.193:0
--local_ip_for_outbound_sockets=127.28.235.193
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39573
--builtin_ntp_servers=127.28.235.212:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:23:37.490620 16942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:37.490893 16942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:37.490934 16942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:37.495085 16942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:37.495144 16942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:37.495229 16942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.193
I20260926 07:23:37.500821 16942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.235.193
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.193
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:37.502031 16942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:37.503508 16942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:37.513468 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:37.513482 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:37.513571 16942 server_base.cc:1034] running on GCE node
W20260926 07:23:37.513494 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:37.514228 16942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:37.514988 16942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:37.516185 16942 hybrid_clock.cc:648] HybridClock initialized: now 1790407417516170 us; error 30 us; skew 500 ppm
I20260926 07:23:37.523602 16942 webserver.cc:460] Webserver started at http://127.28.235.193:45333/ using document root <none> and password file <none>
I20260926 07:23:37.524286 16942 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:37.524405 16942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:37.524672 16942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:37.527031 16942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/data/instance:
uuid: "4372cb69f541457a9af43a2494a93c4d"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "334b3d2123727a96cdaed5baff8f7160"
server_key_iv: "0f7281a2a94a7ac6f121fd5b2d854fc0"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.527609 16942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance:
uuid: "4372cb69f541457a9af43a2494a93c4d"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "334b3d2123727a96cdaed5baff8f7160"
server_key_iv: "0f7281a2a94a7ac6f121fd5b2d854fc0"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.531603 16942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 07:23:37.534394 16956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:37.535681 16942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:23:37.535892 16942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/wal
uuid: "4372cb69f541457a9af43a2494a93c4d"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "334b3d2123727a96cdaed5baff8f7160"
server_key_iv: "0f7281a2a94a7ac6f121fd5b2d854fc0"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.536013 16942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:37.550500 16942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:37.551229 16942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:37.551445 16942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:37.552045 16942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:37.553095 16942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:37.553184 16942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:37.553257 16942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:37.553318 16942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:37.563470 16942 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.193:41273
I20260926 07:23:37.563493 17069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.193:41273 every 8 connection(s)
I20260926 07:23:37.564560 16942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/data/info.pb
I20260926 07:23:37.573331 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 16942
I20260926 07:23:37.573475 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-0/wal/instance
I20260926 07:23:37.573800 29615 external_mini_cluster.cc:1428] Setting key 1961170b095850bce784ff90d5a55b4a
I20260926 07:23:37.574499 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.566593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.193:43683 (local address 127.28.235.254:39573)
0926 07:23:37.566743 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:37.566748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:37.567845 (+ 1097us) server_negotiation.cc:407] Connection header received
0926 07:23:37.569370 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:37.569373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:37.569432 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:37.569513 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:37.570450 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.571014 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.572271 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.572496 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.572593 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.573905 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:37.573914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:37.573917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:37.573930 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:37.573951 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:37.573959 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:37.574045 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:37.574285 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 07:23:37.574355 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:23:37.575488 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.567056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39573 (local address 127.28.235.193:43683)
0926 07:23:37.567691 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:37.567737 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:37.569144 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:37.569672 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:37.569678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:37.569759 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:37.570290 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:37.570302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.571143 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.571146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:37.572071 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:37.572081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.572643 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.572646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:37.572812 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.573621 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:23:37.573623 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:37.573666 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:23:37.573735 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:37.574047 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:37.574060 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:37.574068 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:23:37.574284 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 07:23:37.574653 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260926 07:23:37.575855 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.194:0
--local_ip_for_outbound_sockets=127.28.235.194
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39573
--builtin_ntp_servers=127.28.235.212:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:23:37.576781 17070 heartbeater.cc:347] Connected to a master server at 127.28.235.254:39573
I20260926 07:23:37.577095 17070 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:37.577667 17070 heartbeater.cc:511] Master 127.28.235.254:39573 requested a full tablet report, sending...
I20260926 07:23:37.579367 16882 ts_manager.cc:194] Registered new tserver with Master: 4372cb69f541457a9af43a2494a93c4d (127.28.235.193:41273)
I20260926 07:23:37.580631 16882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.193:43683
W20260926 07:23:37.690328 17073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:37.690644 17073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:37.690712 17073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:37.694938 17073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:37.695034 17073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:37.695127 17073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.194
I20260926 07:23:37.700214 17073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.235.194
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.194
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:37.701462 17073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:37.702919 17073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:37.713153 17079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:37.713160 17081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:37.713554 17073 server_base.cc:1034] running on GCE node
W20260926 07:23:37.713160 17078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:37.714005 17073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:37.714684 17073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:37.715885 17073 hybrid_clock.cc:648] HybridClock initialized: now 1790407417715838 us; error 58 us; skew 500 ppm
I20260926 07:23:37.722397 17073 webserver.cc:460] Webserver started at http://127.28.235.194:39979/ using document root <none> and password file <none>
I20260926 07:23:37.723048 17073 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:37.723146 17073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:37.723393 17073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:37.725679 17073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/data/instance:
uuid: "43412e0f6b2c4a4b996c9d4e756ff417"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "b8ae1491791769940ac54ef846c9cc5a"
server_key_iv: "97cab172e58a097a6a4e3789b4c546e4"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.726248 17073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance:
uuid: "43412e0f6b2c4a4b996c9d4e756ff417"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "b8ae1491791769940ac54ef846c9cc5a"
server_key_iv: "97cab172e58a097a6a4e3789b4c546e4"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.730160 17073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 07:23:37.732897 17087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:37.734354 17073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:37.734508 17073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/wal
uuid: "43412e0f6b2c4a4b996c9d4e756ff417"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "b8ae1491791769940ac54ef846c9cc5a"
server_key_iv: "97cab172e58a097a6a4e3789b4c546e4"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.734658 17073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:37.754734 17073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:37.755486 17073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:37.755689 17073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:37.756295 17073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:37.757515 17073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:37.757596 17073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:37.757670 17073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:37.757719 17073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:37.767657 17073 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.194:37033
I20260926 07:23:37.767673 17200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.194:37033 every 8 connection(s)
I20260926 07:23:37.768606 17073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/data/info.pb
I20260926 07:23:37.773185 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 17073
I20260926 07:23:37.773401 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/wal/instance
I20260926 07:23:37.773697 29615 external_mini_cluster.cc:1428] Setting key 92843ebb533d43be20ef64d26ce3e670
I20260926 07:23:37.775916 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.235.195:0
--local_ip_for_outbound_sockets=127.28.235.195
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39573
--builtin_ntp_servers=127.28.235.212:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:23:37.780092 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.770709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:56983 (local address 127.28.235.254:39573)
0926 07:23:37.770866 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:37.770871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:37.772039 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 07:23:37.773703 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:37.773707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:37.773778 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:37.773871 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:37.774967 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.775627 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.777576 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.777970 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.778160 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.779445 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:37.779452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:37.779458 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:37.779477 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:37.779503 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:37.779516 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:37.779598 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:37.779773 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 07:23:37.779857 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:23:37.780959 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.771104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39573 (local address 127.28.235.194:56983)
0926 07:23:37.771870 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:37.771913 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:37.773448 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:37.774063 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:37.774071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:37.774163 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:37.774785 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:37.774799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.775802 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.775808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:37.777402 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:37.777418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.778193 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.778196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:37.778409 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.779097 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:23:37.779099 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:37.779142 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:23:37.779227 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:37.779616 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:37.779624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:37.779629 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:23:37.779738 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:23:37.780002 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":119,"threads_started":1}
I20260926 07:23:37.782284 17201 heartbeater.cc:347] Connected to a master server at 127.28.235.254:39573
I20260926 07:23:37.782536 17201 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:37.783169 17201 heartbeater.cc:511] Master 127.28.235.254:39573 requested a full tablet report, sending...
I20260926 07:23:37.784371 16882 ts_manager.cc:194] Registered new tserver with Master: 43412e0f6b2c4a4b996c9d4e756ff417 (127.28.235.194:37033)
I20260926 07:23:37.784969 16882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.194:56983
W20260926 07:23:37.905441 17204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:37.905782 17204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:37.905853 17204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:37.910035 17204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:37.910118 17204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:37.910212 17204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.235.195
I20260926 07:23:37.915273 17204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.235.195
--webserver_port=0
--tserver_master_addrs=127.28.235.254:39573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.235.195
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:37.916440 17204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:37.917907 17204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:37.927835 17212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:37.927855 17210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:37.927868 17209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:37.928300 17204 server_base.cc:1034] running on GCE node
I20260926 07:23:37.928766 17204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:37.929507 17204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:37.930714 17204 hybrid_clock.cc:648] HybridClock initialized: now 1790407417930696 us; error 33 us; skew 500 ppm
I20260926 07:23:37.932664 17204 webserver.cc:460] Webserver started at http://127.28.235.195:37843/ using document root <none> and password file <none>
I20260926 07:23:37.933368 17204 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:37.933439 17204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:37.933661 17204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:37.935906 17204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/data/instance:
uuid: "ca98e0d168814af8978a78caae75b1ff"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "f06cb91914ca5448c05859cd0142525a"
server_key_iv: "5814f2e9e03b136f243339ce01b9c395"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.936493 17204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance:
uuid: "ca98e0d168814af8978a78caae75b1ff"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "f06cb91914ca5448c05859cd0142525a"
server_key_iv: "5814f2e9e03b136f243339ce01b9c395"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.940680 17204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 07:23:37.943447 17218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:37.944789 17204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:23:37.945104 17204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/wal
uuid: "ca98e0d168814af8978a78caae75b1ff"
format_stamp: "Formatted at 2026-09-26 07:23:37 on dist-test-slave-7zlw"
server_key: "f06cb91914ca5448c05859cd0142525a"
server_key_iv: "5814f2e9e03b136f243339ce01b9c395"
server_key_version: "encryptionkey@0"
I20260926 07:23:37.945242 17204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:37.968505 17204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:37.969383 17204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:37.969622 17204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:37.970286 17204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:23:37.971274 17204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:23:37.971347 17204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:37.971416 17204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:23:37.971467 17204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:37.981369 17204 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.195:38321
I20260926 07:23:37.981401 17331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.195:38321 every 8 connection(s)
I20260926 07:23:37.982323 17204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/data/info.pb
I20260926 07:23:37.983822 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 17204
I20260926 07:23:37.983978 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/wal/instance
I20260926 07:23:37.984344 29615 external_mini_cluster.cc:1428] Setting key da4693333ee07e62ea7273e72b687870
I20260926 07:23:37.992689 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.984434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.195:52091 (local address 127.28.235.254:39573)
0926 07:23:37.984597 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:37.984602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:37.985985 (+ 1383us) server_negotiation.cc:407] Connection header received
0926 07:23:37.987530 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:37.987533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:37.987578 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:37.987650 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:37.988584 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.989203 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.990484 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.990696 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.990781 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.992219 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:37.992222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:37.992224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:37.992237 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:37.992256 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:37.992263 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:37.992341 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:37.992510 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 07:23:37.992568 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:23:37.993654 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:37.984945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:39573 (local address 127.28.235.195:52091)
0926 07:23:37.985852 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:37.985908 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:37.987322 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:37.987872 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:37.987879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:37.987964 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:23:37.988454 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:37.988467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.989336 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.989338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:37.990340 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:37.990351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:37.990874 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:37.990876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:37.991033 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:37.991886 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:23:37.991887 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:23:37.991930 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:23:37.992001 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:23:37.992374 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:23:37.992382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:23:37.992386 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:23:37.992493 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:23:37.992727 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":129,"threads_started":1}
I20260926 07:23:37.994904 17332 heartbeater.cc:347] Connected to a master server at 127.28.235.254:39573
I20260926 07:23:37.995147 17332 heartbeater.cc:464] Registering TS with master...
I20260926 07:23:37.995738 17332 heartbeater.cc:511] Master 127.28.235.254:39573 requested a full tablet report, sending...
I20260926 07:23:37.996817 16882 ts_manager.cc:194] Registered new tserver with Master: ca98e0d168814af8978a78caae75b1ff (127.28.235.195:38321)
I20260926 07:23:37.997596 16882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.235.195:52091
I20260926 07:23:37.998387 29615 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:23:38.004199 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.000306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.28.235.254:39573)
0926 07:23:38.000445 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:38.000452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:38.000663 (+ 211us) server_negotiation.cc:407] Connection header received
0926 07:23:38.000751 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:38.000753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:38.000790 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:38.000852 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:38.001146 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.001687 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.002546 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.002784 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.002874 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.003731 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:38.003735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:38.003737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:38.003750 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:38.003778 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:38.003786 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:38.003855 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:38.003940 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 07:23:38.003991 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:23:38.007184 16882 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:23:38.010429 16882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:23:38.025197 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.019303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:41273 (local address 127.0.0.1:59084)
0926 07:23:38.020345 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:38.020359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:38.020510 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:38.021195 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:38.021197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:38.021217 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:38.021355 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:38.021363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.022402 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.022406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:38.023649 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:38.023659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.024602 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.024608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:38.024716 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.024758 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 07:23:38.024877 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:23:38.025001 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":28,"thread_start_us":100,"threads_started":1}
I20260926 07:23:38.025243 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.019410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.28.235.193:41273)
0926 07:23:38.020178 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:38.020191 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:38.020437 (+ 246us) server_negotiation.cc:407] Connection header received
0926 07:23:38.020667 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:38.020675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:38.020876 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:38.020975 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:38.021482 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.022275 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.023782 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.024475 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.024600 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.024798 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:38.024924 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 07:23:38.025070 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":131,"threads_started":1}
I20260926 07:23:38.025489 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.019283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.194:37033 (local address 127.0.0.1:33740)
0926 07:23:38.019662 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:38.019717 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:38.019871 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:38.021127 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:38.021136 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:38.021163 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:38.021355 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:38.021363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.022869 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.022872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:38.023994 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:38.024006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.025079 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.025083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:38.025176 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.025200 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:23:38.025270 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 07:23:38.025344 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":97,"thread_start_us":85,"threads_started":1}
I20260926 07:23:38.026237 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.020021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:38321 (local address 127.0.0.1:43160)
0926 07:23:38.020575 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:38.020588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:38.020657 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:38.021297 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:38.021300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:38.021319 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:38.021438 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:38.021448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.022869 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.022872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:38.024507 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:38.024516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.025831 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.025834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:38.025929 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.025951 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:23:38.026022 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:23:38.026092 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260926 07:23:38.026523 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.020199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.28.235.194:37033)
0926 07:23:38.020627 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:38.020640 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:38.020663 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:23:38.020743 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:38.020749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:38.020944 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:38.021124 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:38.021515 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.022736 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.024131 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.024913 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.025873 (+ 960us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.026031 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:38.026078 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 07:23:38.026170 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":116,"threads_started":1}
I20260926 07:23:38.026754 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.020424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.28.235.195:38321)
0926 07:23:38.020868 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:38.020878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:38.020898 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:23:38.020965 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:38.020969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:38.021155 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:38.021284 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:38.021570 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.022731 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.024667 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.025478 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.025604 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.025771 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:38.026465 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 07:23:38.026576 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":142,"threads_started":1}
I20260926 07:23:38.028523 17004 tablet_service.cc:1467] Processing CreateTablet for tablet 7abfad486472466d9b794fc3dd23c63d (DEFAULT_TABLE table=test-table [id=2b4e36044199449fbc44355804ad02cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:38.029764 17004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abfad486472466d9b794fc3dd23c63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:38.029457 17252 tablet_service.cc:1467] Processing CreateTablet for tablet 7abfad486472466d9b794fc3dd23c63d (DEFAULT_TABLE table=test-table [id=2b4e36044199449fbc44355804ad02cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:38.030411 17252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abfad486472466d9b794fc3dd23c63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:38.031165 17135 tablet_service.cc:1467] Processing CreateTablet for tablet 7abfad486472466d9b794fc3dd23c63d (DEFAULT_TABLE table=test-table [id=2b4e36044199449fbc44355804ad02cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:23:38.031970 17135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abfad486472466d9b794fc3dd23c63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:38.037462 17349 tablet_bootstrap.cc:492] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: Bootstrap starting.
I20260926 07:23:38.038780 17351 tablet_bootstrap.cc:492] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: Bootstrap starting.
I20260926 07:23:38.039381 17350 tablet_bootstrap.cc:492] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d: Bootstrap starting.
I20260926 07:23:38.040084 17349 tablet_bootstrap.cc:654] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:38.040982 17351 tablet_bootstrap.cc:654] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:38.041527 17350 tablet_bootstrap.cc:654] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:38.041582 17349 log.cc:824] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:38.041857 17351 log.cc:824] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:38.042317 17350 log.cc:824] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:38.043591 17349 tablet_bootstrap.cc:492] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: No bootstrap required, opened a new log
I20260926 07:23:38.043821 17349 ts_tablet_manager.cc:1397] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:23:38.044041 17350 tablet_bootstrap.cc:492] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d: No bootstrap required, opened a new log
I20260926 07:23:38.044230 17350 ts_tablet_manager.cc:1397] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:23:38.046453 17351 tablet_bootstrap.cc:492] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: No bootstrap required, opened a new log
I20260926 07:23:38.046649 17351 ts_tablet_manager.cc:1397] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 07:23:38.047551 17350 raft_consensus.cc:348] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 } } peers { permanent_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37033 } } peers { permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 } }
I20260926 07:23:38.047816 17350 raft_consensus.cc:374] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:38.047869 17350 raft_consensus.cc:729] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372cb69f541457a9af43a2494a93c4d, State: Initialized, Role: FOLLOWER
I20260926 07:23:38.048071 17349 raft_consensus.cc:348] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 } } peers { permanent_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37033 } } peers { permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 } }
I20260926 07:23:38.048316 17349 raft_consensus.cc:374] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:38.048378 17349 raft_consensus.cc:729] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca98e0d168814af8978a78caae75b1ff, State: Initialized, Role: FOLLOWER
I20260926 07:23:38.048377 17350 consensus_queue.cc:260] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 } } peers { permanent_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37033 } } peers { permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 } }
I20260926 07:23:38.048842 17349 consensus_queue.cc:260] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 } } peers { permanent_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37033 } } peers { permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 } }
I20260926 07:23:38.049566 17350 ts_tablet_manager.cc:1428] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 07:23:38.049649 17070 heartbeater.cc:503] Master 127.28.235.254:39573 was elected leader, sending a full tablet report...
I20260926 07:23:38.050139 17332 heartbeater.cc:503] Master 127.28.235.254:39573 was elected leader, sending a full tablet report...
I20260926 07:23:38.050467 17349 ts_tablet_manager.cc:1428] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:23:38.051424 17351 raft_consensus.cc:348] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 } } peers { permanent_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37033 } } peers { permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 } }
I20260926 07:23:38.051679 17351 raft_consensus.cc:374] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:38.051736 17351 raft_consensus.cc:729] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43412e0f6b2c4a4b996c9d4e756ff417, State: Initialized, Role: FOLLOWER
I20260926 07:23:38.052153 17351 consensus_queue.cc:260] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 } } peers { permanent_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37033 } } peers { permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 } }
I20260926 07:23:38.052958 17201 heartbeater.cc:503] Master 127.28.235.254:39573 was elected leader, sending a full tablet report...
I20260926 07:23:38.053190 17351 ts_tablet_manager.cc:1428] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260926 07:23:38.065891 17071 tablet.cc:2369] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:23:38.153528 17357 raft_consensus.cc:482] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:23:38.153824 17357 raft_consensus.cc:504] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 } } peers { permanent_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37033 } } peers { permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 } }
I20260926 07:23:38.155045 17357 leader_election.cc:288] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372cb69f541457a9af43a2494a93c4d (127.28.235.193:41273), ca98e0d168814af8978a78caae75b1ff (127.28.235.195:38321)
I20260926 07:23:38.158990 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.155397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:41273 (local address 127.28.235.194:58903)
0926 07:23:38.155517 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:38.155534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:38.155639 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:38.155940 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:38.155943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:38.155961 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:38.156087 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:38.156094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.156850 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.156854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:38.157847 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:38.157857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.158513 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.158517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:38.158665 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.158684 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:23:38.158776 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:23:38.158858 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 07:23:38.158990 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.155522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:58903 (local address 127.28.235.193:41273)
0926 07:23:38.155671 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:38.155676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:38.155692 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:23:38.155755 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:38.155758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:38.155812 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:38.155927 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:38.156220 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.156727 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.157977 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.158384 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.158518 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.158577 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:38.158793 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 07:23:38.158863 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:23:38.159201 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.155410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.195:38321 (local address 127.28.235.194:50301)
0926 07:23:38.155895 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:38.155905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:23:38.156024 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:23:38.156299 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:23:38.156302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:23:38.156317 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:23:38.156449 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:38.156455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.157259 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.157265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:38.158080 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:23:38.158089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.158766 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.158770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:23:38.158871 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.158901 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:23:38.158987 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 07:23:38.159054 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":122,"thread_start_us":149,"threads_started":1}
I20260926 07:23:38.159186 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.155605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.235.194:50301 (local address 127.28.235.195:38321)
0926 07:23:38.155716 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:38.155722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:38.156001 (+ 279us) server_negotiation.cc:407] Connection header received
0926 07:23:38.156156 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:38.156159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:38.156200 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:38.156287 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:23:38.156558 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.157145 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.158207 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.158640 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.158777 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.158824 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:38.159002 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 07:23:38.159073 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:23:38.159911 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abfad486472466d9b794fc3dd23c63d" candidate_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca98e0d168814af8978a78caae75b1ff" is_pre_election: true
I20260926 07:23:38.159919 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abfad486472466d9b794fc3dd23c63d" candidate_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372cb69f541457a9af43a2494a93c4d" is_pre_election: true
I20260926 07:23:38.160224 17286 raft_consensus.cc:2454] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43412e0f6b2c4a4b996c9d4e756ff417 in term 0.
I20260926 07:23:38.160240 17016 raft_consensus.cc:2454] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43412e0f6b2c4a4b996c9d4e756ff417 in term 0.
I20260926 07:23:38.160758 17089 leader_election.cc:302] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43412e0f6b2c4a4b996c9d4e756ff417, 4372cb69f541457a9af43a2494a93c4d; no voters:
I20260926 07:23:38.161079 17357 raft_consensus.cc:2781] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:23:38.161178 17357 raft_consensus.cc:482] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:23:38.161257 17357 raft_consensus.cc:3037] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:38.162376 17357 raft_consensus.cc:504] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 } } peers { permanent_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37033 } } peers { permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 } }
I20260926 07:23:38.162729 17357 leader_election.cc:288] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [CANDIDATE]: Term 1 election: Requested vote from peers 4372cb69f541457a9af43a2494a93c4d (127.28.235.193:41273), ca98e0d168814af8978a78caae75b1ff (127.28.235.195:38321)
I20260926 07:23:38.163226 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abfad486472466d9b794fc3dd23c63d" candidate_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372cb69f541457a9af43a2494a93c4d"
I20260926 07:23:38.163226 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abfad486472466d9b794fc3dd23c63d" candidate_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca98e0d168814af8978a78caae75b1ff"
I20260926 07:23:38.163378 17286 raft_consensus.cc:3037] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:38.163378 17016 raft_consensus.cc:3037] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:38.164777 17016 raft_consensus.cc:2454] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43412e0f6b2c4a4b996c9d4e756ff417 in term 1.
I20260926 07:23:38.164777 17286 raft_consensus.cc:2454] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43412e0f6b2c4a4b996c9d4e756ff417 in term 1.
I20260926 07:23:38.165156 17089 leader_election.cc:302] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43412e0f6b2c4a4b996c9d4e756ff417, 4372cb69f541457a9af43a2494a93c4d, ca98e0d168814af8978a78caae75b1ff; no voters:
I20260926 07:23:38.165410 17357 raft_consensus.cc:2781] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:38.165624 17357 raft_consensus.cc:686] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 1 LEADER]: Becoming Leader. State: Replica: 43412e0f6b2c4a4b996c9d4e756ff417, State: Running, Role: LEADER
I20260926 07:23:38.165979 17357 consensus_queue.cc:237] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 } } peers { permanent_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37033 } } peers { permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 } }
I20260926 07:23:38.169571 16880 catalog_manager.cc:5495] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43412e0f6b2c4a4b996c9d4e756ff417 (127.28.235.194). New cstate: current_term: 1 leader_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43412e0f6b2c4a4b996c9d4e756ff417" member_type: VOTER last_known_addr { host: "127.28.235.194" port: 37033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20260926 07:23:38.217614 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.213686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.28.235.194:37033)
0926 07:23:38.213839 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:38.213843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:38.213858 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:23:38.213961 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:38.213966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:38.214047 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:38.214211 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:23:38.214528 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.215108 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.216068 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:38.216276 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:38.216393 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:38.216733 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:23:38.217189 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:23:38.217277 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:38.217426 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 07:23:38.217479 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:23:38.223934 17016 raft_consensus.cc:1261] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d [term 1 FOLLOWER]: Refusing update from remote peer 43412e0f6b2c4a4b996c9d4e756ff417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:38.223934 17286 raft_consensus.cc:1261] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff [term 1 FOLLOWER]: Refusing update from remote peer 43412e0f6b2c4a4b996c9d4e756ff417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:23:38.224715 17359 consensus_queue.cc:1035] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca98e0d168814af8978a78caae75b1ff" member_type: VOTER last_known_addr { host: "127.28.235.195" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:38.224967 17357 consensus_queue.cc:1035] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372cb69f541457a9af43a2494a93c4d" member_type: VOTER last_known_addr { host: "127.28.235.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:23:38.231959 17362 mvcc.cc:205] Tried to move back new op lower bound from 7333508785037275136 to 7333508784811114496. Current Snapshot: MvccSnapshot[applied={T|T < 7333508785037275136}]
I20260926 07:23:38.233860 17364 mvcc.cc:205] Tried to move back new op lower bound from 7333508785037275136 to 7333508784811114496. Current Snapshot: MvccSnapshot[applied={T|T < 7333508785037275136}]
I20260926 07:23:38.234159 17363 mvcc.cc:205] Tried to move back new op lower bound from 7333508785037275136 to 7333508784811114496. Current Snapshot: MvccSnapshot[applied={T|T < 7333508785037275136}]
I20260926 07:23:38.245200 16880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:38.245373 16880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:23:38.247961 16880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9a8b70463914e0ebba376a2cf9609ba: Sending DeleteTablet for 3 replicas of tablet 7abfad486472466d9b794fc3dd23c63d
I20260926 07:23:38.248973 17252 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abfad486472466d9b794fc3dd23c63d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:38 UTC) from {username='slave'} at 127.0.0.1:43160
I20260926 07:23:38.248972 17004 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abfad486472466d9b794fc3dd23c63d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:38 UTC) from {username='slave'} at 127.0.0.1:59084
I20260926 07:23:38.249572 17135 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abfad486472466d9b794fc3dd23c63d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:23:38 UTC) from {username='slave'} at 127.0.0.1:33740
I20260926 07:23:38.249751 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 16942
I20260926 07:23:38.250041 17372 tablet_replica.cc:330] T 7abfad486472466d9b794fc3dd23c63d P 4372cb69f541457a9af43a2494a93c4d: stopping tablet replica
I20260926 07:23:38.251066 17373 tablet_replica.cc:330] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: stopping tablet replica
I20260926 07:23:38.251413 17373 raft_consensus.cc:2229] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:23:38.251871 17373 raft_consensus.cc:2258] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:38.252408 17371 tablet_replica.cc:330] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: stopping tablet replica
I20260926 07:23:38.252736 17371 raft_consensus.cc:2229] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:23:38.253116 17371 raft_consensus.cc:2258] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:23:38.253909 17373 ts_tablet_manager.cc:1905] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:38.254572 17371 ts_tablet_manager.cc:1905] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:23:38.257300 17371 ts_tablet_manager.cc:1918] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:38.257534 17371 log.cc:1196] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-2/wal/wals/7abfad486472466d9b794fc3dd23c63d
I20260926 07:23:38.257903 17371 ts_tablet_manager.cc:1939] T 7abfad486472466d9b794fc3dd23c63d P ca98e0d168814af8978a78caae75b1ff: Deleting consensus metadata
I20260926 07:23:38.257887 17373 ts_tablet_manager.cc:1918] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:23:38.258133 17373 log.cc:1196] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: Deleting WAL directory at /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407154183638-29615-0/minicluster-data/ts-1/wal/wals/7abfad486472466d9b794fc3dd23c63d
I20260926 07:23:38.258514 17373 ts_tablet_manager.cc:1939] T 7abfad486472466d9b794fc3dd23c63d P 43412e0f6b2c4a4b996c9d4e756ff417: Deleting consensus metadata
W20260926 07:23:38.258563 16867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.235.193:41273 (error 108)
I20260926 07:23:38.259519 16866 catalog_manager.cc:4841] TS ca98e0d168814af8978a78caae75b1ff (127.28.235.195:38321): tablet 7abfad486472466d9b794fc3dd23c63d (table test-table [id=2b4e36044199449fbc44355804ad02cc]) successfully deleted
I20260926 07:23:38.259723 16866 catalog_manager.cc:4841] TS 43412e0f6b2c4a4b996c9d4e756ff417 (127.28.235.194:37033): tablet 7abfad486472466d9b794fc3dd23c63d (table test-table [id=2b4e36044199449fbc44355804ad02cc]) successfully deleted
I20260926 07:23:38.259994 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 17073
I20260926 07:23:38.260023 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:38.259733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.193:41273 (local address 127.0.0.1:59094)
0926 07:23:38.259862 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:23:38.259949 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.193:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 07:23:38.260221 16867 catalog_manager.cc:4584] TS 4372cb69f541457a9af43a2494a93c4d (127.28.235.193:41273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7abfad486472466d9b794fc3dd23c63d: Network error: Client connection negotiation failed: client connection to 127.28.235.193:41273: connect: Connection refused (error 111)
I20260926 07:23:38.268347 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 17204
I20260926 07:23:38.274297 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 16851
2026-09-26T07:23:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1181 ms)
[----------] 6 tests from EncryptionPolicyTest (11073 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:38 dist-test-slave-7zlw krb5kdc[17377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:38 dist-test-slave-7zlw krb5kdc[17377](info): set up 2 sockets
Sep 26 07:23:38 dist-test-slave-7zlw krb5kdc[17377](info): commencing operation
krb5kdc: starting...
W20260926 07:23:39.325542 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:39 dist-test-slave-7zlw krb5kdc[17377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:39.456565 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:39.569602 17390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:39.569980 17390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:39.570073 17390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:39.574678 17390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:39.574877 17390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:39.574954 17390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:39.574994 17390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:39.575026 17390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:39.581637 17390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:39.582901 17390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:39.584008 17390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:23:39.593369 17390 server_base.cc:1034] running on GCE node
W20260926 07:23:39.593305 17395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:39.593302 17396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:39.593312 17398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:39.594214 17390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:39.595489 17390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:39.596689 17390 hybrid_clock.cc:648] HybridClock initialized: now 1790407419596669 us; error 36 us; skew 500 ppm
Sep 26 07:23:39 dist-test-slave-7zlw krb5kdc[17377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:39.600010 17390 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:39.601004 17390 webserver.cc:460] Webserver started at http://127.0.0.1:43737/ using document root <none> and password file <none>
I20260926 07:23:39.601634 17390 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:39.601694 17390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:39.601863 17390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:39.603929 17390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "810fa055dc074215aea0ee9ca431ed24"
format_stamp: "Formatted at 2026-09-26 07:23:39 on dist-test-slave-7zlw"
server_key: "e52724c4ddfe8512a07a24468b832500"
server_key_iv: "f8179962db8bf6648d8ccced1b225c62"
server_key_version: "encryptionkey@0"
I20260926 07:23:39.604453 17390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "810fa055dc074215aea0ee9ca431ed24"
format_stamp: "Formatted at 2026-09-26 07:23:39 on dist-test-slave-7zlw"
server_key: "e52724c4ddfe8512a07a24468b832500"
server_key_iv: "f8179962db8bf6648d8ccced1b225c62"
server_key_version: "encryptionkey@0"
I20260926 07:23:39.608316 17390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:23:39.610993 17405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:39.612293 17390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:23:39.612449 17390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "810fa055dc074215aea0ee9ca431ed24"
format_stamp: "Formatted at 2026-09-26 07:23:39 on dist-test-slave-7zlw"
server_key: "e52724c4ddfe8512a07a24468b832500"
server_key_iv: "f8179962db8bf6648d8ccced1b225c62"
server_key_version: "encryptionkey@0"
I20260926 07:23:39.612572 17390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:39.645252 17390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:39.648653 17390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:39.648875 17390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:39.656486 17390 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38135
I20260926 07:23:39.656507 17457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38135 every 8 connection(s)
I20260926 07:23:39.657508 17390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:39.660284 17458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:39.663882 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 17390
I20260926 07:23:39.664016 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:39.664311 29615 external_mini_cluster.cc:1428] Setting key cf0d0eeef7d4af388a500e6ca1a90f2a
I20260926 07:23:39.666527 17458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24: Bootstrap starting.
I20260926 07:23:39.668722 17458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:39.669514 17458 log.cc:824] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:39.671460 17458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24: No bootstrap required, opened a new log
Sep 26 07:23:39 dist-test-slave-7zlw krb5kdc[17377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:23:39.675472 17458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fa055dc074215aea0ee9ca431ed24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38135 } }
I20260926 07:23:39.675740 17458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:39.675820 17458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810fa055dc074215aea0ee9ca431ed24, State: Initialized, Role: FOLLOWER
I20260926 07:23:39.676321 17458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fa055dc074215aea0ee9ca431ed24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38135 } }
I20260926 07:23:39.676477 17458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:39.676573 17458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:39.676681 17458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:39.677848 17458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fa055dc074215aea0ee9ca431ed24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38135 } }
I20260926 07:23:39.678224 17458 leader_election.cc:302] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 810fa055dc074215aea0ee9ca431ed24; no voters:
I20260926 07:23:39.678550 17458 leader_election.cc:288] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:39.678680 17463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:39.678905 17463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [term 1 LEADER]: Becoming Leader. State: Replica: 810fa055dc074215aea0ee9ca431ed24, State: Running, Role: LEADER
I20260926 07:23:39.679142 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:39.665825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43541 (local address 127.0.0.1:38135)
0926 07:23:39.666259 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:39.666274 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:39.666310 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:23:39.667021 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:39.667038 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:39.667409 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:39.667762 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:39.668543 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:39.669672 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:39.670758 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:39.671217 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:39.671384 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:39.674029 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:39.674034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:39.674047 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:39.674075 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:39.675786 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:39.676328 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:39.676334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:39.676338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:39.676446 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:39.676882 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:39.676885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:39.676886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:39.677337 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:39.677500 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:39.677852 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 07:23:39.678096 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":130,"threads_started":1}
I20260926 07:23:39.679257 17463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fa055dc074215aea0ee9ca431ed24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38135 } }
I20260926 07:23:39.679949 17458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:39.680445 17463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 810fa055dc074215aea0ee9ca431ed24. Latest consensus state: current_term: 1 leader_uuid: "810fa055dc074215aea0ee9ca431ed24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fa055dc074215aea0ee9ca431ed24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38135 } } }
I20260926 07:23:39.680557 17463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:39.681568 17464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "810fa055dc074215aea0ee9ca431ed24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810fa055dc074215aea0ee9ca431ed24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38135 } } }
I20260926 07:23:39.681748 17464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:39.682061 17468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:39.682336 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:39.686738 17468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:39.692873 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:39.684384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52887 (local address 127.0.0.1:38135)
0926 07:23:39.684491 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:39.684497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:39.684709 (+ 212us) server_negotiation.cc:407] Connection header received
0926 07:23:39.684860 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:39.684863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:39.684911 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:39.684992 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:39.685660 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:39.686174 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:39.687165 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:39.687419 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:39.687532 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:39.689102 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:39.689107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:39.689111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:39.689138 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:39.690869 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:39.691431 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:39.691435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:39.691437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:39.691522 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:39.691881 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:39.691885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:39.691887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:39.692107 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:39.692246 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:39.692591 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 07:23:39.692692 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 07:23:39.694049 17468 catalog_manager.cc:1347] Generated new cluster ID: 00f4c8713e5a4d64b98e30840f645c42
I20260926 07:23:39.694110 17468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:39.705960 17468 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:39.706943 17468 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:39.711241 17468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 810fa055dc074215aea0ee9ca431ed24: Generated new TSK 0
I20260926 07:23:39.711970 17468 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:39.717778 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 17390
W20260926 07:23:39.723592 17375 connection.cc:535] client connection to 127.0.0.1:38135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38135): Connection reset by peer (error 104)
2026-09-26T07:23:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:39 dist-test-slave-7zlw krb5kdc[17489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:39 dist-test-slave-7zlw krb5kdc[17489](info): set up 2 sockets
Sep 26 07:23:39 dist-test-slave-7zlw krb5kdc[17489](info): commencing operation
krb5kdc: starting...
W20260926 07:23:40.761765 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:40 dist-test-slave-7zlw krb5kdc[17489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:40.898202 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:41.020231 17502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:41.020538 17502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:41.020587 17502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:41.024910 17502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:41.024987 17502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:41.025077 17502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:41.025120 17502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:41.025157 17502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:41.030706 17502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:41.031775 17502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:41.032753 17502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:41.042351 17508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:41.042297 17510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:41.042405 17502 server_base.cc:1034] running on GCE node
W20260926 07:23:41.042328 17507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:41.043078 17502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:41.044201 17502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:41.045426 17502 hybrid_clock.cc:648] HybridClock initialized: now 1790407421045398 us; error 78 us; skew 500 ppm
Sep 26 07:23:41 dist-test-slave-7zlw krb5kdc[17489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:41.048295 17502 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:41.049295 17502 webserver.cc:460] Webserver started at http://127.0.0.1:44869/ using document root <none> and password file <none>
I20260926 07:23:41.049826 17502 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:41.049885 17502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:41.050051 17502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:41.052104 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "b7fec5e1156e45d3a4ef72b469613204"
format_stamp: "Formatted at 2026-09-26 07:23:41 on dist-test-slave-7zlw"
server_key: "9de675979d60c0baf4fb0ede22c4b3ff"
server_key_iv: "fd20293229e3e40cee3d0f9e3dfc6f70"
server_key_version: "encryptionkey@0"
I20260926 07:23:41.052655 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "b7fec5e1156e45d3a4ef72b469613204"
format_stamp: "Formatted at 2026-09-26 07:23:41 on dist-test-slave-7zlw"
server_key: "9de675979d60c0baf4fb0ede22c4b3ff"
server_key_iv: "fd20293229e3e40cee3d0f9e3dfc6f70"
server_key_version: "encryptionkey@0"
I20260926 07:23:41.056572 17502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:23:41.059118 17517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:41.060537 17502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:23:41.060693 17502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "b7fec5e1156e45d3a4ef72b469613204"
format_stamp: "Formatted at 2026-09-26 07:23:41 on dist-test-slave-7zlw"
server_key: "9de675979d60c0baf4fb0ede22c4b3ff"
server_key_iv: "fd20293229e3e40cee3d0f9e3dfc6f70"
server_key_version: "encryptionkey@0"
I20260926 07:23:41.060804 17502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:41.081364 17502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:41.084867 17502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:41.085134 17502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:41.092180 17502 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39723
I20260926 07:23:41.092263 17569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39723 every 8 connection(s)
I20260926 07:23:41.093232 17502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:41.095510 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 17502
I20260926 07:23:41.095654 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:41.095968 29615 external_mini_cluster.cc:1428] Setting key b7cc5fbdb74aea90ded124f408ee99d5
I20260926 07:23:41.096335 17570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:41.103384 17570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204: Bootstrap starting.
I20260926 07:23:41.105274 17573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:41.097437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52863 (local address 127.0.0.1:39723)
0926 07:23:41.097884 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:41.097900 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:41.097936 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:23:41.098570 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:41.098585 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:41.098917 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:41.099196 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:41.099579 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:41.100590 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:41.101820 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:41.102300 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:41.102427 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:41.103423 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:41.103428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:41.103440 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:41.103900 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:41.103953 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:41.103976 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:41.104077 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:41.104284 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 07:23:41.104447 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":166,"threads_started":1}
I20260926 07:23:41.105844 17570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:41.106694 17570 log.cc:824] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:41.108470 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:41.108943 17570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204: No bootstrap required, opened a new log
I20260926 07:23:41.113756 17570 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fec5e1156e45d3a4ef72b469613204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39723 } }
I20260926 07:23:41.114077 17570 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:41.114145 17570 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fec5e1156e45d3a4ef72b469613204, State: Initialized, Role: FOLLOWER
I20260926 07:23:41.114681 17570 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fec5e1156e45d3a4ef72b469613204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39723 } }
I20260926 07:23:41.114840 17570 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:41.114974 17570 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:41.115101 17570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:41.115270 17573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:41.110741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42019 (local address 127.0.0.1:39723)
0926 07:23:41.110868 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:41.110872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:41.110988 (+ 116us) server_negotiation.cc:407] Connection header received
0926 07:23:41.111150 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:41.111152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:41.111191 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:41.111266 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:41.111631 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:41.112147 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:41.113424 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:41.113655 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:41.113743 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:41.114749 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:41.114752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:41.114754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:41.114765 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:41.114790 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:41.114800 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:41.114922 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:41.115092 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 07:23:41.115132 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:23:41.116338 17570 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fec5e1156e45d3a4ef72b469613204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39723 } }
I20260926 07:23:41.116719 17570 leader_election.cc:302] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7fec5e1156e45d3a4ef72b469613204; no voters:
I20260926 07:23:41.117093 17570 leader_election.cc:288] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:41.117214 17580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:41.117615 17580 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [term 1 LEADER]: Becoming Leader. State: Replica: b7fec5e1156e45d3a4ef72b469613204, State: Running, Role: LEADER
I20260926 07:23:41.118029 17580 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fec5e1156e45d3a4ef72b469613204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39723 } }
I20260926 07:23:41.118527 17570 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:41.119843 17582 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7fec5e1156e45d3a4ef72b469613204. Latest consensus state: current_term: 1 leader_uuid: "b7fec5e1156e45d3a4ef72b469613204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fec5e1156e45d3a4ef72b469613204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39723 } } }
I20260926 07:23:41.119846 17581 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7fec5e1156e45d3a4ef72b469613204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7fec5e1156e45d3a4ef72b469613204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39723 } } }
I20260926 07:23:41.120000 17582 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:41.120012 17581 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204 [sys.catalog]: This master's current role is: LEADER
W20260926 07:23:41.123579 17595 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:23:41.123662 17595 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:23:41.123732 17591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:41.124617 17591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:41.130452 17591 catalog_manager.cc:1347] Generated new cluster ID: bfb0668200d04ef4bc267bdd74238357
I20260926 07:23:41.130528 17591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:41.137506 17591 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:41.138855 17591 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:41.146559 17591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7fec5e1156e45d3a4ef72b469613204: Generated new TSK 0
I20260926 07:23:41.147439 17591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:41.155373 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 17502
W20260926 07:23:41.161067 17487 connection.cc:535] client connection to 127.0.0.1:39723 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39723): Connection reset by peer (error 104)
2026-09-26T07:23:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:41 dist-test-slave-7zlw krb5kdc[17601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:41 dist-test-slave-7zlw krb5kdc[17601](info): set up 2 sockets
Sep 26 07:23:41 dist-test-slave-7zlw krb5kdc[17601](info): commencing operation
krb5kdc: starting...
W20260926 07:23:42.209435 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:42 dist-test-slave-7zlw krb5kdc[17601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:42.337155 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:42.449398 17614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:42.449747 17614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:42.449831 17614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:42.453914 17614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:42.454023 17614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:42.454061 17614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:42.454118 17614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:42.454175 17614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:42.459702 17614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36409
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:42.460934 17614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:42.462014 17614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:42.471167 17620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:42.471153 17619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:42.471331 17614 server_base.cc:1034] running on GCE node
W20260926 07:23:42.471153 17622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:42.472087 17614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:42.473328 17614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:42.474536 17614 hybrid_clock.cc:648] HybridClock initialized: now 1790407422474499 us; error 47 us; skew 500 ppm
Sep 26 07:23:42 dist-test-slave-7zlw krb5kdc[17601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:42.477620 17614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:42.478549 17614 webserver.cc:460] Webserver started at http://127.0.0.1:36223/ using document root <none> and password file <none>
I20260926 07:23:42.479135 17614 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:42.479245 17614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:42.479436 17614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:42.481580 17614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "58fd2662c9854fa491d02d79a57fd169"
format_stamp: "Formatted at 2026-09-26 07:23:42 on dist-test-slave-7zlw"
server_key: "c45dc0f40fd07ac47e791c6c0e80ea9e"
server_key_iv: "7952d0e6d8757bb483805f3fa39956d6"
server_key_version: "encryptionkey@0"
I20260926 07:23:42.482115 17614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "58fd2662c9854fa491d02d79a57fd169"
format_stamp: "Formatted at 2026-09-26 07:23:42 on dist-test-slave-7zlw"
server_key: "c45dc0f40fd07ac47e791c6c0e80ea9e"
server_key_iv: "7952d0e6d8757bb483805f3fa39956d6"
server_key_version: "encryptionkey@0"
I20260926 07:23:42.486006 17614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:23:42.488487 17629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:42.489897 17614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:42.490008 17614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "58fd2662c9854fa491d02d79a57fd169"
format_stamp: "Formatted at 2026-09-26 07:23:42 on dist-test-slave-7zlw"
server_key: "c45dc0f40fd07ac47e791c6c0e80ea9e"
server_key_iv: "7952d0e6d8757bb483805f3fa39956d6"
server_key_version: "encryptionkey@0"
I20260926 07:23:42.490095 17614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:42.494773 17614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:42.498044 17614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:42.498214 17614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:42.505579 17614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36409
I20260926 07:23:42.505601 17681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36409 every 8 connection(s)
I20260926 07:23:42.506515 17614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:42.509279 17682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:42.513887 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 17614
I20260926 07:23:42.514034 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:42.514343 29615 external_mini_cluster.cc:1428] Setting key ee77eade25fa50ee5453364624aac0b4
I20260926 07:23:42.515111 17682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169: Bootstrap starting.
I20260926 07:23:42.517581 17682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:42.518465 17682 log.cc:824] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:42.519717 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:42.515812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50419 (local address 127.0.0.1:36409)
0926 07:23:42.516250 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:42.516269 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:42.516306 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:23:42.516926 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:42.516942 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:42.517307 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:42.517636 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:42.518431 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:42.518436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:42.518448 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:42.518459 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:42.518509 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:42.518529 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:42.518647 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:42.518779 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:23:42.518936 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":131,"threads_started":1}
I20260926 07:23:42.520532 17682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169: No bootstrap required, opened a new log
I20260926 07:23:42.522523 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:42.524591 17682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fd2662c9854fa491d02d79a57fd169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } }
I20260926 07:23:42.524920 17682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:42.524991 17682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58fd2662c9854fa491d02d79a57fd169, State: Initialized, Role: FOLLOWER
I20260926 07:23:42.525571 17682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fd2662c9854fa491d02d79a57fd169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } }
I20260926 07:23:42.525772 17682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:42.525856 17682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:42.525949 17682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:42.526868 17682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fd2662c9854fa491d02d79a57fd169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } }
I20260926 07:23:42.527173 17682 leader_election.cc:302] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58fd2662c9854fa491d02d79a57fd169; no voters:
I20260926 07:23:42.527527 17682 leader_election.cc:288] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:42.527904 17693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:42.528085 17693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [term 1 LEADER]: Becoming Leader. State: Replica: 58fd2662c9854fa491d02d79a57fd169, State: Running, Role: LEADER
I20260926 07:23:42.528388 17693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fd2662c9854fa491d02d79a57fd169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } }
I20260926 07:23:42.528966 17682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:42.529712 17694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58fd2662c9854fa491d02d79a57fd169. Latest consensus state: current_term: 1 leader_uuid: "58fd2662c9854fa491d02d79a57fd169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fd2662c9854fa491d02d79a57fd169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } } }
I20260926 07:23:42.529879 17694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:42.530072 17692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58fd2662c9854fa491d02d79a57fd169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fd2662c9854fa491d02d79a57fd169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } } }
I20260926 07:23:42.530166 17692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169 [sys.catalog]: This master's current role is: LEADER
W20260926 07:23:42.533744 17707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:23:42.533838 17707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:23:42.533931 17700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:42.534614 17700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:42.537992 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:42.524540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49341 (local address 127.0.0.1:36409)
0926 07:23:42.524715 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:42.524720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:42.524873 (+ 153us) server_negotiation.cc:407] Connection header received
0926 07:23:42.525062 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:42.525065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:42.525125 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:42.525214 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:42.537188 (+ 11974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:42.537192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:42.537227 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:42.537246 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:42.537279 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:42.537319 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:42.537432 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:42.537822 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 07:23:42.537867 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:23:42.540832 17700 catalog_manager.cc:1347] Generated new cluster ID: 1064e28868fc4fc49f60aa930de86d45
I20260926 07:23:42.540900 17700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:42.562517 17700 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:42.562944 17700 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:42.567004 17700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58fd2662c9854fa491d02d79a57fd169: Generated new TSK 0
I20260926 07:23:42.567749 17700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:42.575277 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 17614
2026-09-26T07:23:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:42 dist-test-slave-7zlw krb5kdc[17713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:42 dist-test-slave-7zlw krb5kdc[17713](info): set up 2 sockets
Sep 26 07:23:42 dist-test-slave-7zlw krb5kdc[17713](info): commencing operation
krb5kdc: starting...
W20260926 07:23:43.637320 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:43 dist-test-slave-7zlw krb5kdc[17713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:43.772933 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:43.886897 17726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:43.887207 17726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:43.887252 17726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:43.891556 17726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:43.891633 17726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:43.891654 17726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:43.891670 17726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:43.891686 17726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:43.897625 17726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:43.898836 17726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:43.899848 17726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:43.908980 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:43.908964 17731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:43.909271 17726 server_base.cc:1034] running on GCE node
W20260926 07:23:43.909407 17732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:43.910032 17726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:43.911258 17726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:43.912460 17726 hybrid_clock.cc:648] HybridClock initialized: now 1790407423912423 us; error 55 us; skew 500 ppm
Sep 26 07:23:43 dist-test-slave-7zlw krb5kdc[17713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:43.915769 17726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:43.916718 17726 webserver.cc:460] Webserver started at http://127.0.0.1:33591/ using document root <none> and password file <none>
I20260926 07:23:43.917388 17726 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:43.917459 17726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:43.917634 17726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:43.919759 17726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "963f2f5e17e04a7a9496bb1571a0ae3b"
format_stamp: "Formatted at 2026-09-26 07:23:43 on dist-test-slave-7zlw"
server_key: "adb2ac3cf6ce79091585522cbde6204c"
server_key_iv: "b67330c38f10e159d8d282aec47b72e0"
server_key_version: "encryptionkey@0"
I20260926 07:23:43.920305 17726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "963f2f5e17e04a7a9496bb1571a0ae3b"
format_stamp: "Formatted at 2026-09-26 07:23:43 on dist-test-slave-7zlw"
server_key: "adb2ac3cf6ce79091585522cbde6204c"
server_key_iv: "b67330c38f10e159d8d282aec47b72e0"
server_key_version: "encryptionkey@0"
I20260926 07:23:43.924420 17726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:23:43.927215 17741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:43.928696 17726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:23:43.928814 17726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "963f2f5e17e04a7a9496bb1571a0ae3b"
format_stamp: "Formatted at 2026-09-26 07:23:43 on dist-test-slave-7zlw"
server_key: "adb2ac3cf6ce79091585522cbde6204c"
server_key_iv: "b67330c38f10e159d8d282aec47b72e0"
server_key_version: "encryptionkey@0"
I20260926 07:23:43.928934 17726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:43.952992 17726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:43.956627 17726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:43.956882 17726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:43.965230 17726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38997
I20260926 07:23:43.965288 17793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38997 every 8 connection(s)
I20260926 07:23:43.966279 17726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:43.969340 17794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:43.970270 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 17726
I20260926 07:23:43.970448 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:43.970793 29615 external_mini_cluster.cc:1428] Setting key 87988616dce453233faf780697cc0a66
I20260926 07:23:43.976374 17794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b: Bootstrap starting.
I20260926 07:23:43.979560 17794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:43.980644 17794 log.cc:824] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:23:43 dist-test-slave-7zlw krb5kdc[17713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:23:43.982928 17794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b: No bootstrap required, opened a new log
I20260926 07:23:43.986992 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:43.972484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59257 (local address 127.0.0.1:38997)
0926 07:23:43.972958 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:43.972974 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:43.973010 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:23:43.973770 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:43.973786 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:43.974128 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:43.974503 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:43.975441 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:43.976422 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:43.977772 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:43.978233 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:43.978355 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:43.981791 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:43.981796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:43.981812 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:43.981841 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:43.983751 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:43.984285 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:43.984292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:43.984298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:43.984462 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:43.984972 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:43.984976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:43.984977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:43.985474 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:43.985705 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:43.985969 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 07:23:43.986270 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
I20260926 07:23:43.987659 17794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963f2f5e17e04a7a9496bb1571a0ae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } }
I20260926 07:23:43.987948 17794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:43.988003 17794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963f2f5e17e04a7a9496bb1571a0ae3b, State: Initialized, Role: FOLLOWER
I20260926 07:23:43.988436 17794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963f2f5e17e04a7a9496bb1571a0ae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } }
I20260926 07:23:43.988600 17794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:43.988701 17794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:43.988828 17794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:43.990200 17794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963f2f5e17e04a7a9496bb1571a0ae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } }
I20260926 07:23:43.990386 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:43.990604 17794 leader_election.cc:302] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 963f2f5e17e04a7a9496bb1571a0ae3b; no voters:
I20260926 07:23:43.990973 17794 leader_election.cc:288] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:43.991097 17799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:43.991328 17799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [term 1 LEADER]: Becoming Leader. State: Replica: 963f2f5e17e04a7a9496bb1571a0ae3b, State: Running, Role: LEADER
I20260926 07:23:43.991714 17799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963f2f5e17e04a7a9496bb1571a0ae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } }
I20260926 07:23:43.992360 17794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:43.993129 17801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 963f2f5e17e04a7a9496bb1571a0ae3b. Latest consensus state: current_term: 1 leader_uuid: "963f2f5e17e04a7a9496bb1571a0ae3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963f2f5e17e04a7a9496bb1571a0ae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } } }
I20260926 07:23:43.993314 17801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:43.993597 17800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "963f2f5e17e04a7a9496bb1571a0ae3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963f2f5e17e04a7a9496bb1571a0ae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } } }
I20260926 07:23:43.993705 17800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:43.994184 17812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:43.997691 17812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:44.003887 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:43.992840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41701 (local address 127.0.0.1:38997)
0926 07:23:43.992956 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:43.992961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:43.993804 (+ 843us) server_negotiation.cc:407] Connection header received
0926 07:23:43.994016 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:43.994020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:43.994075 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:43.994190 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:43.995316 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:43.996142 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:43.997449 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:43.997813 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:43.997947 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:43.999964 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:43.999970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:43.999974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:44.000004 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:44.001783 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:44.002344 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:44.002349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:44.002351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:44.002447 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:44.002785 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:44.002790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:44.002793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:44.003105 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:44.003290 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:44.003552 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 07:23:44.003706 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:23:44.006474 17812 catalog_manager.cc:1347] Generated new cluster ID: c41ef3396691476a840ae1a413b4da46
I20260926 07:23:44.006557 17812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:44.022967 17812 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:44.023994 17812 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:44.031252 17812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 963f2f5e17e04a7a9496bb1571a0ae3b: Generated new TSK 0
I20260926 07:23:44.032009 17812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:44.038702 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 17726
2026-09-26T07:23:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:44 dist-test-slave-7zlw krb5kdc[17825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:44 dist-test-slave-7zlw krb5kdc[17825](info): set up 2 sockets
Sep 26 07:23:44 dist-test-slave-7zlw krb5kdc[17825](info): commencing operation
krb5kdc: starting...
W20260926 07:23:45.093768 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:45 dist-test-slave-7zlw krb5kdc[17825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:45.232471 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:45.352110 17838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:45.352502 17838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:45.352591 17838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:45.356859 17838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:45.356987 17838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:45.357030 17838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:45.357096 17838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:45.357131 17838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:45.363001 17838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:45.364189 17838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:45.365311 17838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:45.374747 17843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:45.374821 17838 server_base.cc:1034] running on GCE node
W20260926 07:23:45.374739 17846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:45.374732 17844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:45.375633 17838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:45.376843 17838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:45.378077 17838 hybrid_clock.cc:648] HybridClock initialized: now 1790407425378070 us; error 50 us; skew 500 ppm
Sep 26 07:23:45 dist-test-slave-7zlw krb5kdc[17825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:45.380952 17838 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:45.381968 17838 webserver.cc:460] Webserver started at http://127.0.0.1:40065/ using document root <none> and password file <none>
I20260926 07:23:45.382550 17838 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:45.382642 17838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:45.382875 17838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:45.384928 17838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "82881e6d7d94492788d83bcb8bf63525"
format_stamp: "Formatted at 2026-09-26 07:23:45 on dist-test-slave-7zlw"
server_key: "67ab59b3775db8fd5d3f62ae3229820f"
server_key_iv: "a2db4032d6f5219a8aa9b3d6b1857513"
server_key_version: "encryptionkey@0"
I20260926 07:23:45.385548 17838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "82881e6d7d94492788d83bcb8bf63525"
format_stamp: "Formatted at 2026-09-26 07:23:45 on dist-test-slave-7zlw"
server_key: "67ab59b3775db8fd5d3f62ae3229820f"
server_key_iv: "a2db4032d6f5219a8aa9b3d6b1857513"
server_key_version: "encryptionkey@0"
I20260926 07:23:45.389451 17838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:23:45.391908 17853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:45.393160 17838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:23:45.393343 17838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "82881e6d7d94492788d83bcb8bf63525"
format_stamp: "Formatted at 2026-09-26 07:23:45 on dist-test-slave-7zlw"
server_key: "67ab59b3775db8fd5d3f62ae3229820f"
server_key_iv: "a2db4032d6f5219a8aa9b3d6b1857513"
server_key_version: "encryptionkey@0"
I20260926 07:23:45.393476 17838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:45.415086 17838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:45.418588 17838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:45.418819 17838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:45.426553 17838 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42247
I20260926 07:23:45.426587 17905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42247 every 8 connection(s)
I20260926 07:23:45.427625 17838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:45.430028 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 17838
I20260926 07:23:45.430147 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:45.430467 29615 external_mini_cluster.cc:1428] Setting key 4d8173995d7792d7771548841803a825
I20260926 07:23:45.430725 17906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:45.438086 17906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525: Bootstrap starting.
I20260926 07:23:45.440909 17906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:45.441320 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:45.431915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38341 (local address 127.0.0.1:42247)
0926 07:23:45.432361 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:45.432379 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:45.432420 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:23:45.433586 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:45.433610 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:45.434083 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:45.434456 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:45.434883 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:45.436338 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:45.437417 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:45.438090 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:45.438256 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:45.439191 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:45.439198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:45.439213 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:45.439730 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:45.439809 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:45.439837 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:45.439942 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:45.440130 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 07:23:45.440341 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
I20260926 07:23:45.441811 17906 log.cc:824] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:45.444214 17906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525: No bootstrap required, opened a new log
I20260926 07:23:45.444597 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:45.448679 17906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82881e6d7d94492788d83bcb8bf63525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42247 } }
I20260926 07:23:45.448952 17906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:45.449018 17906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82881e6d7d94492788d83bcb8bf63525, State: Initialized, Role: FOLLOWER
I20260926 07:23:45.449527 17906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82881e6d7d94492788d83bcb8bf63525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42247 } }
I20260926 07:23:45.449688 17906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:45.449788 17906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:45.449918 17906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:45.450896 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:45.446856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57631 (local address 127.0.0.1:42247)
0926 07:23:45.446984 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:45.446988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:45.447107 (+ 119us) server_negotiation.cc:407] Connection header received
0926 07:23:45.447269 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:45.447272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:45.447324 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:45.447408 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:45.447693 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:45.448228 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:45.449180 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:45.449445 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:45.449564 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:45.450449 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:45.450453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:45.450456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:45.450469 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:45.450501 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:45.450511 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:45.450589 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:45.450701 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 07:23:45.450756 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:23:45.451028 17906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82881e6d7d94492788d83bcb8bf63525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42247 } }
I20260926 07:23:45.451452 17906 leader_election.cc:302] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82881e6d7d94492788d83bcb8bf63525; no voters:
I20260926 07:23:45.451836 17906 leader_election.cc:288] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:45.451957 17916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:45.452180 17916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [term 1 LEADER]: Becoming Leader. State: Replica: 82881e6d7d94492788d83bcb8bf63525, State: Running, Role: LEADER
I20260926 07:23:45.452493 17916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82881e6d7d94492788d83bcb8bf63525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42247 } }
I20260926 07:23:45.453228 17906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:45.454252 17918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82881e6d7d94492788d83bcb8bf63525. Latest consensus state: current_term: 1 leader_uuid: "82881e6d7d94492788d83bcb8bf63525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82881e6d7d94492788d83bcb8bf63525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42247 } } }
I20260926 07:23:45.454393 17918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:45.454692 17917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82881e6d7d94492788d83bcb8bf63525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82881e6d7d94492788d83bcb8bf63525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42247 } } }
I20260926 07:23:45.454790 17917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:45.454799 17925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:45.457998 17925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:45.464021 17925 catalog_manager.cc:1347] Generated new cluster ID: c6fb44e5f1114fef8c9c94fc4cf7e488
I20260926 07:23:45.464107 17925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:45.485940 17925 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:45.487433 17925 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:45.497865 17925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82881e6d7d94492788d83bcb8bf63525: Generated new TSK 0
I20260926 07:23:45.498719 17925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:45.512887 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 17838
2026-09-26T07:23:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1474 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:45 dist-test-slave-7zlw krb5kdc[17937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:45 dist-test-slave-7zlw krb5kdc[17937](info): set up 2 sockets
Sep 26 07:23:45 dist-test-slave-7zlw krb5kdc[17937](info): commencing operation
krb5kdc: starting...
W20260926 07:23:46.589433 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:46 dist-test-slave-7zlw krb5kdc[17937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:46.716953 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:46.830139 17950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:46.830497 17950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:46.830587 17950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:46.835005 17950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:46.835108 17950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:46.835162 17950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:46.835222 17950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:46.835260 17950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:46.840940 17950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:46.842192 17950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:46.843250 17950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:46.852139 17958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:46.852152 17955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:46.852322 17950 server_base.cc:1034] running on GCE node
W20260926 07:23:46.852216 17956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:46.853119 17950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:46.854422 17950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:46.855633 17950 hybrid_clock.cc:648] HybridClock initialized: now 1790407426855600 us; error 47 us; skew 500 ppm
Sep 26 07:23:46 dist-test-slave-7zlw krb5kdc[17937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:46.858671 17950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:46.859647 17950 webserver.cc:460] Webserver started at http://127.0.0.1:42541/ using document root <none> and password file <none>
I20260926 07:23:46.860216 17950 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:46.860317 17950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:46.860549 17950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:46.862697 17950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "bf378e3a137f4cf5a42fe463026d632c"
format_stamp: "Formatted at 2026-09-26 07:23:46 on dist-test-slave-7zlw"
server_key: "869e04340d4f07e1150bb26b9f7ce1fc"
server_key_iv: "bcee20eaa7779917da616c7381c95f87"
server_key_version: "encryptionkey@0"
I20260926 07:23:46.863262 17950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "bf378e3a137f4cf5a42fe463026d632c"
format_stamp: "Formatted at 2026-09-26 07:23:46 on dist-test-slave-7zlw"
server_key: "869e04340d4f07e1150bb26b9f7ce1fc"
server_key_iv: "bcee20eaa7779917da616c7381c95f87"
server_key_version: "encryptionkey@0"
I20260926 07:23:46.867251 17950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:23:46.869976 17965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:46.871345 17950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:46.871513 17950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "bf378e3a137f4cf5a42fe463026d632c"
format_stamp: "Formatted at 2026-09-26 07:23:46 on dist-test-slave-7zlw"
server_key: "869e04340d4f07e1150bb26b9f7ce1fc"
server_key_iv: "bcee20eaa7779917da616c7381c95f87"
server_key_version: "encryptionkey@0"
I20260926 07:23:46.871651 17950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:46.876293 17950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:46.879509 17950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:46.879711 17950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:46.886603 17950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33273
I20260926 07:23:46.886688 18017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33273 every 8 connection(s)
I20260926 07:23:46.887605 17950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:46.890722 18018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:46.893723 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 17950
I20260926 07:23:46.893910 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:46.894193 29615 external_mini_cluster.cc:1428] Setting key acb42e1e27652dcb3f219841b556cbd6
I20260926 07:23:46.896730 18018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c: Bootstrap starting.
I20260926 07:23:46.899286 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:46.895487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47433 (local address 127.0.0.1:33273)
0926 07:23:46.895905 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:46.895920 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:46.895956 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:23:46.896582 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:46.896597 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:46.896914 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:46.897237 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:46.898072 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:46.898077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:46.898088 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:46.898098 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:46.898148 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:46.898166 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:46.898269 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:46.898408 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 07:23:46.898590 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":143,"threads_started":1}
I20260926 07:23:46.899365 18018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:46.900276 18018 log.cc:824] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:46.902107 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:46.902314 18018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c: No bootstrap required, opened a new log
I20260926 07:23:46.906086 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:46.904417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51801 (local address 127.0.0.1:33273)
0926 07:23:46.904530 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:46.904534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:46.904657 (+ 123us) server_negotiation.cc:407] Connection header received
0926 07:23:46.904784 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:46.904826 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:46.904865 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:46.904963 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:46.905644 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:46.905647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:46.905649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:46.905658 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:46.905681 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:46.905691 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:46.905759 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:46.905885 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 07:23:46.905918 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:23:46.907208 18018 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf378e3a137f4cf5a42fe463026d632c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260926 07:23:46.907428 18018 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:46.907505 18018 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf378e3a137f4cf5a42fe463026d632c, State: Initialized, Role: FOLLOWER
I20260926 07:23:46.907925 18018 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf378e3a137f4cf5a42fe463026d632c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260926 07:23:46.908061 18018 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:46.908123 18018 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:46.908227 18018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:46.909247 18018 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf378e3a137f4cf5a42fe463026d632c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260926 07:23:46.909548 18018 leader_election.cc:302] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf378e3a137f4cf5a42fe463026d632c; no voters:
I20260926 07:23:46.909893 18018 leader_election.cc:288] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:46.910022 18028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:46.910307 18028 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [term 1 LEADER]: Becoming Leader. State: Replica: bf378e3a137f4cf5a42fe463026d632c, State: Running, Role: LEADER
I20260926 07:23:46.910640 18028 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf378e3a137f4cf5a42fe463026d632c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260926 07:23:46.911151 18018 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:46.911792 18028 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf378e3a137f4cf5a42fe463026d632c. Latest consensus state: current_term: 1 leader_uuid: "bf378e3a137f4cf5a42fe463026d632c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf378e3a137f4cf5a42fe463026d632c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } } }
I20260926 07:23:46.911976 18028 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:46.912204 18029 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf378e3a137f4cf5a42fe463026d632c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf378e3a137f4cf5a42fe463026d632c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } } }
I20260926 07:23:46.912328 18029 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:46.912489 18037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:46.915688 18037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:46.922102 18037 catalog_manager.cc:1347] Generated new cluster ID: 5a7a07de55ea4217b695a0b2be3fd24b
I20260926 07:23:46.922178 18037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:46.931851 18037 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:46.932206 18037 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:46.947341 18037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf378e3a137f4cf5a42fe463026d632c: Generated new TSK 0
I20260926 07:23:46.948333 18037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:46.957253 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 17950
2026-09-26T07:23:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:47 dist-test-slave-7zlw krb5kdc[18049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:47 dist-test-slave-7zlw krb5kdc[18049](info): set up 2 sockets
Sep 26 07:23:47 dist-test-slave-7zlw krb5kdc[18049](info): commencing operation
krb5kdc: starting...
W20260926 07:23:48.029318 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:48 dist-test-slave-7zlw krb5kdc[18049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:48.159852 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:48.273910 18062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:48.274236 18062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:48.274283 18062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:48.278716 18062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:48.278800 18062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:48.278820 18062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:48.278837 18062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:48.278853 18062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:48.284508 18062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:48.285825 18062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:48.287408 18062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:48.297134 18070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:48.297219 18062 server_base.cc:1034] running on GCE node
W20260926 07:23:48.297134 18068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:48.297163 18067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:48.298012 18062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:48.299192 18062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:48.300447 18062 hybrid_clock.cc:648] HybridClock initialized: now 1790407428300414 us; error 46 us; skew 500 ppm
Sep 26 07:23:48 dist-test-slave-7zlw krb5kdc[18049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:48.303615 18062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:48.304672 18062 webserver.cc:460] Webserver started at http://127.0.0.1:37429/ using document root <none> and password file <none>
I20260926 07:23:48.305374 18062 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:48.305469 18062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:48.305702 18062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:48.307889 18062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "4b206a40bb7e45109199af743efe89b3"
format_stamp: "Formatted at 2026-09-26 07:23:48 on dist-test-slave-7zlw"
server_key: "4b08d44f5142dfa5cde72f5bb35f651f"
server_key_iv: "6b8ebd7c90d86f42f142fa7f21730865"
server_key_version: "encryptionkey@0"
I20260926 07:23:48.308471 18062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "4b206a40bb7e45109199af743efe89b3"
format_stamp: "Formatted at 2026-09-26 07:23:48 on dist-test-slave-7zlw"
server_key: "4b08d44f5142dfa5cde72f5bb35f651f"
server_key_iv: "6b8ebd7c90d86f42f142fa7f21730865"
server_key_version: "encryptionkey@0"
I20260926 07:23:48.312662 18062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:23:48.316021 18077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:48.317658 18062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 07:23:48.317814 18062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "4b206a40bb7e45109199af743efe89b3"
format_stamp: "Formatted at 2026-09-26 07:23:48 on dist-test-slave-7zlw"
server_key: "4b08d44f5142dfa5cde72f5bb35f651f"
server_key_iv: "6b8ebd7c90d86f42f142fa7f21730865"
server_key_version: "encryptionkey@0"
I20260926 07:23:48.317955 18062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:48.333099 18062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:48.336715 18062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:48.336937 18062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:48.344990 18062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41475
I20260926 07:23:48.345043 18129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41475 every 8 connection(s)
I20260926 07:23:48.346001 18062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:48.346818 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 18062
I20260926 07:23:48.346961 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:48.347280 29615 external_mini_cluster.cc:1428] Setting key 6122fe657b68f58fe7cd057199754f35
I20260926 07:23:48.349519 18130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:48.355283 18130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3: Bootstrap starting.
Sep 26 07:23:48 dist-test-slave-7zlw krb5kdc[18049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:23:48.357760 18130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:48.358628 18130 log.cc:824] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:48.361132 18130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3: No bootstrap required, opened a new log
I20260926 07:23:48.362898 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:48.349061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:57553 (local address 127.0.0.1:41475)
0926 07:23:48.349594 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:48.349617 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:48.349665 (+ 48us) server_negotiation.cc:407] Connection header received
0926 07:23:48.350324 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:48.350340 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:48.350682 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:48.351023 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:48.351871 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:48.353117 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:48.354020 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:48.354544 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:48.354726 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:48.357319 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:48.357324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:48.357339 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:48.357368 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:48.359388 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:48.360057 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:48.360062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:48.360067 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:48.360202 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:48.360595 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:48.360598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:48.360599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:48.361128 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:48.361426 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:48.361809 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 07:23:48.362107 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":152,"threads_started":1}
I20260926 07:23:48.365357 18130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b206a40bb7e45109199af743efe89b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } }
I20260926 07:23:48.365577 18130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:48.365645 18130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b206a40bb7e45109199af743efe89b3, State: Initialized, Role: FOLLOWER
I20260926 07:23:48.366077 18130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b206a40bb7e45109199af743efe89b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } }
I20260926 07:23:48.366225 18130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:48.366317 18130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:48.366362 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:48.366418 18130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:48.367707 18130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b206a40bb7e45109199af743efe89b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } }
I20260926 07:23:48.368104 18130 leader_election.cc:302] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b206a40bb7e45109199af743efe89b3; no voters:
I20260926 07:23:48.368481 18130 leader_election.cc:288] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:48.368583 18139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:48.368852 18139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [term 1 LEADER]: Becoming Leader. State: Replica: 4b206a40bb7e45109199af743efe89b3, State: Running, Role: LEADER
I20260926 07:23:48.369310 18139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b206a40bb7e45109199af743efe89b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } }
I20260926 07:23:48.369966 18130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:48.371026 18140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b206a40bb7e45109199af743efe89b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b206a40bb7e45109199af743efe89b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } } }
I20260926 07:23:48.371165 18140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:48.371371 18140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b206a40bb7e45109199af743efe89b3. Latest consensus state: current_term: 1 leader_uuid: "4b206a40bb7e45109199af743efe89b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b206a40bb7e45109199af743efe89b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } } }
I20260926 07:23:48.371444 18140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3 [sys.catalog]: This master's current role is: LEADER
W20260926 07:23:48.375834 18155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:23:48.375938 18155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:23:48.376021 18149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:48.376909 18149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:48.378697 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:48.369104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:33003 (local address 127.0.0.1:41475)
0926 07:23:48.369236 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:48.369241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:48.369406 (+ 165us) server_negotiation.cc:407] Connection header received
0926 07:23:48.369597 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:48.369600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:48.369654 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:48.369744 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:48.370596 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:48.371448 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:48.372639 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:48.372975 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:48.373107 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:48.374969 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:48.374973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:48.374976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:48.375003 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:48.376714 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:48.377319 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:48.377324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:48.377326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:48.377457 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:48.377811 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:48.377815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:48.377816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:48.377997 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:48.378156 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:48.378422 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 07:23:48.378521 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:23:48.382899 18149 catalog_manager.cc:1347] Generated new cluster ID: 271867ada33f46a4aca13af5e9774ac3
I20260926 07:23:48.382977 18149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:48.400688 18149 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:48.401813 18149 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:48.410749 18149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b206a40bb7e45109199af743efe89b3: Generated new TSK 0
I20260926 07:23:48.411645 18149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:48.425132 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 18062
W20260926 07:23:48.431048 18047 connection.cc:535] client connection to 127.0.0.1:41475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41475): Connection reset by peer (error 104)
2026-09-26T07:23:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:48 dist-test-slave-7zlw krb5kdc[18161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:48 dist-test-slave-7zlw krb5kdc[18161](info): set up 2 sockets
Sep 26 07:23:48 dist-test-slave-7zlw krb5kdc[18161](info): commencing operation
krb5kdc: starting...
W20260926 07:23:49.477396 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:49 dist-test-slave-7zlw krb5kdc[18161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:49.605456 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:49.718837 18174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:49.719126 18174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:49.719169 18174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:49.723345 18174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:49.723425 18174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:49.723445 18174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:49.723464 18174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:49.723479 18174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:49.729207 18174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:49.730463 18174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:49.731554 18174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:49.740490 18179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:49.740556 18174 server_base.cc:1034] running on GCE node
W20260926 07:23:49.740501 18180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:49.740501 18182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:49.741468 18174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:49.742679 18174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:49.743887 18174 hybrid_clock.cc:648] HybridClock initialized: now 1790407429743868 us; error 32 us; skew 500 ppm
Sep 26 07:23:49 dist-test-slave-7zlw krb5kdc[18161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:49.746974 18174 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:49.748075 18174 webserver.cc:460] Webserver started at http://127.0.0.1:36927/ using document root <none> and password file <none>
I20260926 07:23:49.748667 18174 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:49.748762 18174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:49.749065 18174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:49.751325 18174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "f416745757da46b8a7b011e144de53a3"
format_stamp: "Formatted at 2026-09-26 07:23:49 on dist-test-slave-7zlw"
server_key: "55fc5aa1d0000f8a7d1c516a41653d0b"
server_key_iv: "d635a6efd4cfd99116dd422704f6c245"
server_key_version: "encryptionkey@0"
I20260926 07:23:49.751942 18174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "f416745757da46b8a7b011e144de53a3"
format_stamp: "Formatted at 2026-09-26 07:23:49 on dist-test-slave-7zlw"
server_key: "55fc5aa1d0000f8a7d1c516a41653d0b"
server_key_iv: "d635a6efd4cfd99116dd422704f6c245"
server_key_version: "encryptionkey@0"
I20260926 07:23:49.755918 18174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:23:49.758852 18189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:49.760222 18174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:23:49.760393 18174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "f416745757da46b8a7b011e144de53a3"
format_stamp: "Formatted at 2026-09-26 07:23:49 on dist-test-slave-7zlw"
server_key: "55fc5aa1d0000f8a7d1c516a41653d0b"
server_key_iv: "d635a6efd4cfd99116dd422704f6c245"
server_key_version: "encryptionkey@0"
I20260926 07:23:49.760536 18174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:49.777310 18174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:49.780864 18174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:49.781152 18174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:49.788419 18174 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42529
I20260926 07:23:49.788467 18241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42529 every 8 connection(s)
I20260926 07:23:49.789614 18174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:49.792868 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 18174
I20260926 07:23:49.793064 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:49.793131 18242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:49.793447 29615 external_mini_cluster.cc:1428] Setting key 7fd6708bfa2a25a057367b406b4f1721
I20260926 07:23:49.801343 18242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3: Bootstrap starting.
I20260926 07:23:49.803714 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:49.794945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:46695 (local address 127.0.0.1:42529)
0926 07:23:49.795432 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:49.795458 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:49.795495 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:23:49.796286 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:49.796307 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:49.796711 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:49.797063 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:49.797487 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:49.798561 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:49.799856 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:49.800537 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:49.800669 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:49.801830 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:49.801835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:49.801849 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:49.802274 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:49.802342 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:49.802367 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:49.802480 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:49.802722 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 07:23:49.802918 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":182,"threads_started":1}
I20260926 07:23:49.803968 18242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:49.804817 18242 log.cc:824] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:49.807297 18242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3: No bootstrap required, opened a new log
I20260926 07:23:49.807320 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:49.812465 18242 raft_consensus.cc:348] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f416745757da46b8a7b011e144de53a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } }
I20260926 07:23:49.812769 18242 raft_consensus.cc:374] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:49.812809 18242 raft_consensus.cc:729] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f416745757da46b8a7b011e144de53a3, State: Initialized, Role: FOLLOWER
I20260926 07:23:49.813344 18242 consensus_queue.cc:260] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f416745757da46b8a7b011e144de53a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } }
I20260926 07:23:49.813486 18242 raft_consensus.cc:388] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:49.813606 18242 raft_consensus.cc:482] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:49.813728 18242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:49.814531 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:49.809767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:52677 (local address 127.0.0.1:42529)
0926 07:23:49.809966 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:49.809972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:49.810289 (+ 317us) server_negotiation.cc:407] Connection header received
0926 07:23:49.810451 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:49.810453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:49.810497 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:49.810566 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:49.810873 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:49.811403 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:49.812505 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:49.812790 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:49.812899 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:49.814004 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:49.814009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:49.814018 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:49.814032 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:49.814060 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:49.814071 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:49.814150 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:49.814323 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 07:23:49.814382 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 07:23:49.815048 18242 raft_consensus.cc:504] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f416745757da46b8a7b011e144de53a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } }
I20260926 07:23:49.815451 18242 leader_election.cc:302] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f416745757da46b8a7b011e144de53a3; no voters:
I20260926 07:23:49.815819 18242 leader_election.cc:288] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:49.815944 18252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:49.816217 18252 raft_consensus.cc:686] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [term 1 LEADER]: Becoming Leader. State: Replica: f416745757da46b8a7b011e144de53a3, State: Running, Role: LEADER
I20260926 07:23:49.816576 18252 consensus_queue.cc:237] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f416745757da46b8a7b011e144de53a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } }
I20260926 07:23:49.817397 18242 sys_catalog.cc:564] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:49.818346 18253 sys_catalog.cc:455] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f416745757da46b8a7b011e144de53a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f416745757da46b8a7b011e144de53a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } } }
I20260926 07:23:49.818483 18253 sys_catalog.cc:458] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:49.819008 18261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:49.819245 18254 sys_catalog.cc:455] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f416745757da46b8a7b011e144de53a3. Latest consensus state: current_term: 1 leader_uuid: "f416745757da46b8a7b011e144de53a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f416745757da46b8a7b011e144de53a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42529 } } }
I20260926 07:23:49.819353 18254 sys_catalog.cc:458] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:49.822575 18261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:49.828408 18261 catalog_manager.cc:1347] Generated new cluster ID: 4f60ab2ebca94e6c8cb7c0f1ef239dca
I20260926 07:23:49.828497 18261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:49.841921 18261 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:49.843269 18261 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:49.856784 18261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f416745757da46b8a7b011e144de53a3: Generated new TSK 0
I20260926 07:23:49.857805 18261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:49.863965 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 18174
W20260926 07:23:49.870138 18159 connection.cc:535] client connection to 127.0.0.1:42529 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42529): Connection reset by peer (error 104)
2026-09-26T07:23:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:49 dist-test-slave-7zlw krb5kdc[18273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:49 dist-test-slave-7zlw krb5kdc[18273](info): set up 2 sockets
Sep 26 07:23:49 dist-test-slave-7zlw krb5kdc[18273](info): commencing operation
krb5kdc: starting...
W20260926 07:23:50.917305 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:51 dist-test-slave-7zlw krb5kdc[18273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:51.050673 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:51.163009 18286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:51.163319 18286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:51.163363 18286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:51.167626 18286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:51.167706 18286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:51.167726 18286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:51.167744 18286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:51.167759 18286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:51.173480 18286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:51.174853 18286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:51.175941 18286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:51.184911 18294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:51.184930 18292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:51.185113 18286 server_base.cc:1034] running on GCE node
W20260926 07:23:51.184911 18291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:51.185875 18286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:51.187053 18286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:51.188292 18286 hybrid_clock.cc:648] HybridClock initialized: now 1790407431188272 us; error 32 us; skew 500 ppm
Sep 26 07:23:51 dist-test-slave-7zlw krb5kdc[18273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:51.191347 18286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:51.192327 18286 webserver.cc:460] Webserver started at http://127.0.0.1:36703/ using document root <none> and password file <none>
I20260926 07:23:51.192884 18286 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:51.192945 18286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:51.193188 18286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:51.195387 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "72894a35f1e6449996c8d23492f6a430"
format_stamp: "Formatted at 2026-09-26 07:23:51 on dist-test-slave-7zlw"
server_key: "94a79700869d5f422e4b9512776a0bf5"
server_key_iv: "2370c5578d0ee6226d620aaf23a8ebc0"
server_key_version: "encryptionkey@0"
I20260926 07:23:51.196064 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "72894a35f1e6449996c8d23492f6a430"
format_stamp: "Formatted at 2026-09-26 07:23:51 on dist-test-slave-7zlw"
server_key: "94a79700869d5f422e4b9512776a0bf5"
server_key_iv: "2370c5578d0ee6226d620aaf23a8ebc0"
server_key_version: "encryptionkey@0"
I20260926 07:23:51.199901 18286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:23:51.202499 18301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:51.203814 18286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:23:51.203931 18286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "72894a35f1e6449996c8d23492f6a430"
format_stamp: "Formatted at 2026-09-26 07:23:51 on dist-test-slave-7zlw"
server_key: "94a79700869d5f422e4b9512776a0bf5"
server_key_iv: "2370c5578d0ee6226d620aaf23a8ebc0"
server_key_version: "encryptionkey@0"
I20260926 07:23:51.204031 18286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:51.208953 18286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:51.212397 18286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:51.212553 18286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:51.220070 18286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34663
I20260926 07:23:51.220091 18353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34663 every 8 connection(s)
I20260926 07:23:51.221114 18286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:51.223948 18354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:51.227147 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 18286
I20260926 07:23:51.227303 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:51.227664 29615 external_mini_cluster.cc:1428] Setting key be8dbd2aacb775680461bf385d4021df
I20260926 07:23:51.230212 18354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430: Bootstrap starting.
I20260926 07:23:51.232645 18354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:51.232980 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:51.229185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:53543 (local address 127.0.0.1:34663)
0926 07:23:51.229627 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:51.229642 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:51.229677 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:23:51.230363 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:51.230388 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:51.230665 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:51.230995 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:51.231783 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:51.231787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:51.231799 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:51.231811 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:51.231864 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:51.231883 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:51.232023 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:51.232096 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 07:23:51.232257 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":133,"threads_started":1}
I20260926 07:23:51.233595 18354 log.cc:824] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:51.235602 18354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430: No bootstrap required, opened a new log
I20260926 07:23:51.235776 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:51.239567 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:51.237849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:37779 (local address 127.0.0.1:34663)
0926 07:23:51.237966 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:51.237969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:51.238078 (+ 109us) server_negotiation.cc:407] Connection header received
0926 07:23:51.238201 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:51.238203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:51.238232 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:51.238299 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:51.239041 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:51.239043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:51.239045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:51.239055 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:51.239078 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:51.239089 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:51.239156 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:51.239323 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 07:23:51.239404 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:23:51.239959 18354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72894a35f1e6449996c8d23492f6a430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34663 } }
I20260926 07:23:51.240211 18354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:51.240280 18354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72894a35f1e6449996c8d23492f6a430, State: Initialized, Role: FOLLOWER
I20260926 07:23:51.240690 18354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72894a35f1e6449996c8d23492f6a430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34663 } }
I20260926 07:23:51.240840 18354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:51.240931 18354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:51.241058 18354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:51.242014 18354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72894a35f1e6449996c8d23492f6a430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34663 } }
I20260926 07:23:51.242370 18354 leader_election.cc:302] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72894a35f1e6449996c8d23492f6a430; no voters:
I20260926 07:23:51.242691 18354 leader_election.cc:288] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:51.242923 18364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:51.243232 18364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [term 1 LEADER]: Becoming Leader. State: Replica: 72894a35f1e6449996c8d23492f6a430, State: Running, Role: LEADER
I20260926 07:23:51.243577 18364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72894a35f1e6449996c8d23492f6a430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34663 } }
I20260926 07:23:51.244000 18354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:51.245299 18366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72894a35f1e6449996c8d23492f6a430. Latest consensus state: current_term: 1 leader_uuid: "72894a35f1e6449996c8d23492f6a430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72894a35f1e6449996c8d23492f6a430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34663 } } }
I20260926 07:23:51.245429 18366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:51.245831 18373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:51.245787 18365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72894a35f1e6449996c8d23492f6a430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72894a35f1e6449996c8d23492f6a430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34663 } } }
I20260926 07:23:51.245923 18365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:51.248814 18373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:51.254693 18373 catalog_manager.cc:1347] Generated new cluster ID: ff46e1d605d54dc4a2fe050c559a73b7
I20260926 07:23:51.254776 18373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:51.286098 18373 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:51.286491 18373 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:51.293836 18373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72894a35f1e6449996c8d23492f6a430: Generated new TSK 0
I20260926 07:23:51.294631 18373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:51.308679 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 18286
2026-09-26T07:23:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:51 dist-test-slave-7zlw krb5kdc[18385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:51 dist-test-slave-7zlw krb5kdc[18385](info): set up 2 sockets
Sep 26 07:23:51 dist-test-slave-7zlw krb5kdc[18385](info): commencing operation
krb5kdc: starting...
W20260926 07:23:52.361521 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:52 dist-test-slave-7zlw krb5kdc[18385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:52.488854 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:52.602337 18398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:52.602667 18398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:52.602752 18398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:52.606940 18398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:52.607035 18398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:52.607072 18398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:52.607137 18398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:52.607194 18398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:52.612756 18398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:52.613967 18398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:52.615018 18398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:52.623788 18404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:52.623759 18406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:52.623764 18403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:52.623863 18398 server_base.cc:1034] running on GCE node
I20260926 07:23:52.624550 18398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:52.625747 18398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:52.626922 18398 hybrid_clock.cc:648] HybridClock initialized: now 1790407432626910 us; error 26 us; skew 500 ppm
Sep 26 07:23:52 dist-test-slave-7zlw krb5kdc[18385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:52.629944 18398 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:52.630961 18398 webserver.cc:460] Webserver started at http://127.0.0.1:33823/ using document root <none> and password file <none>
I20260926 07:23:52.631578 18398 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:52.631640 18398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:52.631870 18398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:52.634099 18398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3"
format_stamp: "Formatted at 2026-09-26 07:23:52 on dist-test-slave-7zlw"
server_key: "325e2003d629fc0101da07001bf860cd"
server_key_iv: "23ef737201990f4907046ede0095e2ae"
server_key_version: "encryptionkey@0"
I20260926 07:23:52.634680 18398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3"
format_stamp: "Formatted at 2026-09-26 07:23:52 on dist-test-slave-7zlw"
server_key: "325e2003d629fc0101da07001bf860cd"
server_key_iv: "23ef737201990f4907046ede0095e2ae"
server_key_version: "encryptionkey@0"
I20260926 07:23:52.638520 18398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:23:52.640976 18413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:52.642437 18398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:52.642577 18398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3"
format_stamp: "Formatted at 2026-09-26 07:23:52 on dist-test-slave-7zlw"
server_key: "325e2003d629fc0101da07001bf860cd"
server_key_iv: "23ef737201990f4907046ede0095e2ae"
server_key_version: "encryptionkey@0"
I20260926 07:23:52.642697 18398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:52.670254 18398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:52.673588 18398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:52.673851 18398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:52.681598 18398 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37307
I20260926 07:23:52.681746 18465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37307 every 8 connection(s)
I20260926 07:23:52.682638 18398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:52.685971 18466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:52.685974 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 18398
I20260926 07:23:52.686157 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:52.686440 29615 external_mini_cluster.cc:1428] Setting key 18740a29fc03d62b2bf02d2a31d24ae7
I20260926 07:23:52.694120 18466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3: Bootstrap starting.
I20260926 07:23:52.696802 18466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3: Neither blocks nor log segments found. Creating new log.
Sep 26 07:23:52 dist-test-slave-7zlw krb5kdc[18385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:23:52.697868 18466 log.cc:824] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:52.700548 18466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3: No bootstrap required, opened a new log
I20260926 07:23:52.703343 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:52.687964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:45099 (local address 127.0.0.1:37307)
0926 07:23:52.688387 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:52.688407 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:52.688453 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:23:52.689567 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:52.689594 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:52.690075 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:52.690468 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:52.691654 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:52.693239 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:52.694076 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:52.694759 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:52.694930 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:52.697784 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:52.697790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:52.697804 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:52.697834 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:52.699920 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:52.700594 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:52.700599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:52.700604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:52.700775 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:52.701177 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:52.701180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:52.701181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:52.701658 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:52.701845 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:52.702301 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 07:23:52.702562 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
I20260926 07:23:52.704771 18466 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37307 } }
I20260926 07:23:52.705022 18466 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:52.705122 18466 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d49d2a34da4a1c8fe2b79dfa5dd1a3, State: Initialized, Role: FOLLOWER
I20260926 07:23:52.705658 18466 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37307 } }
I20260926 07:23:52.705825 18466 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:52.705910 18466 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:52.706010 18466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:52.706410 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:52.707082 18466 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37307 } }
I20260926 07:23:52.707454 18466 leader_election.cc:302] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7d49d2a34da4a1c8fe2b79dfa5dd1a3; no voters:
I20260926 07:23:52.707800 18466 leader_election.cc:288] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:52.707875 18474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:52.708088 18474 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [term 1 LEADER]: Becoming Leader. State: Replica: e7d49d2a34da4a1c8fe2b79dfa5dd1a3, State: Running, Role: LEADER
I20260926 07:23:52.708433 18474 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37307 } }
I20260926 07:23:52.709224 18466 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:52.709606 18477 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d49d2a34da4a1c8fe2b79dfa5dd1a3. Latest consensus state: current_term: 1 leader_uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37307 } } }
I20260926 07:23:52.709755 18477 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:52.710234 18481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:52.710484 18476 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d49d2a34da4a1c8fe2b79dfa5dd1a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37307 } } }
I20260926 07:23:52.710656 18476 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:52.713681 18481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:52.719246 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:52.708728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:53129 (local address 127.0.0.1:37307)
0926 07:23:52.708867 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:52.708874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:52.709209 (+ 335us) server_negotiation.cc:407] Connection header received
0926 07:23:52.709373 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:52.709378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:52.709432 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:52.709503 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:52.710518 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:52.711943 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:52.713130 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:52.713453 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:52.713571 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:52.715333 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:52.715336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:52.715339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:52.715366 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:52.717131 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:52.717807 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:52.717811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:52.717813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:52.717902 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:52.718281 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:52.718284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:52.718286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:52.718485 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:52.718650 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:52.718965 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 07:23:52.719070 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":41}
I20260926 07:23:52.721066 18481 catalog_manager.cc:1347] Generated new cluster ID: edc8eee24349469b8bcd716b8903ee48
I20260926 07:23:52.721169 18481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:52.739437 18481 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:52.740491 18481 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:52.745513 18481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7d49d2a34da4a1c8fe2b79dfa5dd1a3: Generated new TSK 0
I20260926 07:23:52.746402 18481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:52.758291 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 18398
W20260926 07:23:52.764081 18383 connection.cc:535] client connection to 127.0.0.1:37307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37307): Connection reset by peer (error 104)
2026-09-26T07:23:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:52 dist-test-slave-7zlw krb5kdc[18497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:52 dist-test-slave-7zlw krb5kdc[18497](info): set up 2 sockets
Sep 26 07:23:52 dist-test-slave-7zlw krb5kdc[18497](info): commencing operation
krb5kdc: starting...
W20260926 07:23:53.825253 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:53 dist-test-slave-7zlw krb5kdc[18497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:53.952430 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:54.065750 18510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:54.066045 18510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:54.066088 18510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:54.070194 18510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:54.070268 18510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:54.070289 18510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:54.070307 18510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:54.070322 18510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:54.075806 18510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45503
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:54.076877 18510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:54.077898 18510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:54.087168 18516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:54.087332 18518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:54.087214 18510 server_base.cc:1034] running on GCE node
W20260926 07:23:54.087227 18515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:54.088045 18510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:54.089403 18510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:54.090606 18510 hybrid_clock.cc:648] HybridClock initialized: now 1790407434090601 us; error 41 us; skew 500 ppm
Sep 26 07:23:54 dist-test-slave-7zlw krb5kdc[18497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:54.105300 18510 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:54.106447 18510 webserver.cc:460] Webserver started at http://127.0.0.1:40617/ using document root <none> and password file <none>
I20260926 07:23:54.107077 18510 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:54.107174 18510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:54.107414 18510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:54.109663 18510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "265e7b788e6740048d38ce228c71f313"
format_stamp: "Formatted at 2026-09-26 07:23:54 on dist-test-slave-7zlw"
server_key: "4c1c0b1e404a9b763290221bc258220c"
server_key_iv: "df0eb1799595c0dcd43d4c0f9d4d52a3"
server_key_version: "encryptionkey@0"
I20260926 07:23:54.110268 18510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "265e7b788e6740048d38ce228c71f313"
format_stamp: "Formatted at 2026-09-26 07:23:54 on dist-test-slave-7zlw"
server_key: "4c1c0b1e404a9b763290221bc258220c"
server_key_iv: "df0eb1799595c0dcd43d4c0f9d4d52a3"
server_key_version: "encryptionkey@0"
I20260926 07:23:54.114311 18510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 07:23:54.116974 18525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:54.118616 18510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:54.118793 18510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "265e7b788e6740048d38ce228c71f313"
format_stamp: "Formatted at 2026-09-26 07:23:54 on dist-test-slave-7zlw"
server_key: "4c1c0b1e404a9b763290221bc258220c"
server_key_iv: "df0eb1799595c0dcd43d4c0f9d4d52a3"
server_key_version: "encryptionkey@0"
I20260926 07:23:54.118933 18510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:54.147183 18510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:54.150374 18510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:54.150589 18510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:54.158006 18510 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45503
I20260926 07:23:54.158030 18577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45503 every 8 connection(s)
I20260926 07:23:54.158977 18510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:54.159405 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 18510
I20260926 07:23:54.159579 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:54.159865 29615 external_mini_cluster.cc:1428] Setting key 663621346a60b15c18ba0831e8720826
I20260926 07:23:54.162225 18578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:54.169206 18578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313: Bootstrap starting.
I20260926 07:23:54.170220 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:54.161409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:34341 (local address 127.0.0.1:45503)
0926 07:23:54.161845 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:54.161864 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:54.161907 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:23:54.162750 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:54.162773 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:54.163187 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:54.163552 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:54.163950 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:54.165487 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:54.166402 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:54.167060 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:54.167231 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:54.168188 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:54.168194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:54.168211 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:54.168713 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:54.168781 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:54.168826 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:54.168956 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:54.169184 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 07:23:54.169333 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":132,"threads_started":1}
I20260926 07:23:54.171931 18578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:54.172796 18578 log.cc:824] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:54.173369 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:54.175382 18578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313: No bootstrap required, opened a new log
I20260926 07:23:54.180091 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:54.175726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:45989 (local address 127.0.0.1:45503)
0926 07:23:54.175845 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:54.175850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:54.176046 (+ 196us) server_negotiation.cc:407] Connection header received
0926 07:23:54.176142 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:54.176145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:54.176192 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:54.176278 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:54.176518 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:54.177326 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:54.178149 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:54.178478 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:54.178586 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:54.179526 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:54.179530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:54.179533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:54.179553 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:54.179583 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:54.179599 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:54.179707 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:54.179910 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 07:23:54.179950 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:23:54.180639 18578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265e7b788e6740048d38ce228c71f313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } }
I20260926 07:23:54.180876 18578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:54.180909 18578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265e7b788e6740048d38ce228c71f313, State: Initialized, Role: FOLLOWER
I20260926 07:23:54.181384 18578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265e7b788e6740048d38ce228c71f313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } }
I20260926 07:23:54.181511 18578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:54.181597 18578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:54.181707 18578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:54.182801 18578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265e7b788e6740048d38ce228c71f313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } }
I20260926 07:23:54.183169 18578 leader_election.cc:302] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 265e7b788e6740048d38ce228c71f313; no voters:
I20260926 07:23:54.183596 18578 leader_election.cc:288] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:54.183750 18588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:54.184000 18588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [term 1 LEADER]: Becoming Leader. State: Replica: 265e7b788e6740048d38ce228c71f313, State: Running, Role: LEADER
I20260926 07:23:54.184398 18588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265e7b788e6740048d38ce228c71f313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } }
I20260926 07:23:54.184921 18578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:54.186045 18590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 265e7b788e6740048d38ce228c71f313. Latest consensus state: current_term: 1 leader_uuid: "265e7b788e6740048d38ce228c71f313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265e7b788e6740048d38ce228c71f313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } } }
I20260926 07:23:54.186180 18590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:54.186208 18589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "265e7b788e6740048d38ce228c71f313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265e7b788e6740048d38ce228c71f313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45503 } } }
I20260926 07:23:54.186317 18589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313 [sys.catalog]: This master's current role is: LEADER
W20260926 07:23:54.189769 18603 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:23:54.189865 18603 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:23:54.189952 18597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:54.190646 18597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:54.196209 18597 catalog_manager.cc:1347] Generated new cluster ID: d6634a4e88e84d1b97a9ab69571018f7
I20260926 07:23:54.196287 18597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:54.209267 18597 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:54.210666 18597 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:54.227916 18597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 265e7b788e6740048d38ce228c71f313: Generated new TSK 0
I20260926 07:23:54.228852 18597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:54.244040 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 18510
W20260926 07:23:54.249804 18495 connection.cc:535] client connection to 127.0.0.1:45503 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45503): Connection reset by peer (error 104)
2026-09-26T07:23:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1485 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:54 dist-test-slave-7zlw krb5kdc[18609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:54 dist-test-slave-7zlw krb5kdc[18609](info): set up 2 sockets
Sep 26 07:23:54 dist-test-slave-7zlw krb5kdc[18609](info): commencing operation
krb5kdc: starting...
W20260926 07:23:55.297511 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:55 dist-test-slave-7zlw krb5kdc[18609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:23:55.426349 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:55.543545 18622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:55.543864 18622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:55.543948 18622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:55.548100 18622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:55.548204 18622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:55.548244 18622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:55.548277 18622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:55.548308 18622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:55.553767 18622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:55.554896 18622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:55.555914 18622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:55.564329 18627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:55.564388 18630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:55.564472 18622 server_base.cc:1034] running on GCE node
W20260926 07:23:55.564348 18628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:55.565274 18622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:55.566358 18622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:55.567515 18622 hybrid_clock.cc:648] HybridClock initialized: now 1790407435567481 us; error 49 us; skew 500 ppm
Sep 26 07:23:55 dist-test-slave-7zlw krb5kdc[18609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:55.570580 18622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:23:55.571535 18622 webserver.cc:460] Webserver started at http://127.0.0.1:44091/ using document root <none> and password file <none>
I20260926 07:23:55.572109 18622 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:55.572170 18622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:55.572379 18622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:55.574558 18622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "f38558bad86040d99cd6831d1315b2d2"
format_stamp: "Formatted at 2026-09-26 07:23:55 on dist-test-slave-7zlw"
server_key: "4ab6dc6a6ec1b2c6c32d821a1d4f6ead"
server_key_iv: "1e942c8e8bf1fb9ef19c3d5fd8f68bfa"
server_key_version: "encryptionkey@0"
I20260926 07:23:55.575135 18622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "f38558bad86040d99cd6831d1315b2d2"
format_stamp: "Formatted at 2026-09-26 07:23:55 on dist-test-slave-7zlw"
server_key: "4ab6dc6a6ec1b2c6c32d821a1d4f6ead"
server_key_iv: "1e942c8e8bf1fb9ef19c3d5fd8f68bfa"
server_key_version: "encryptionkey@0"
I20260926 07:23:55.579056 18622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:23:55.581614 18637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:55.582933 18622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:23:55.583143 18622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "f38558bad86040d99cd6831d1315b2d2"
format_stamp: "Formatted at 2026-09-26 07:23:55 on dist-test-slave-7zlw"
server_key: "4ab6dc6a6ec1b2c6c32d821a1d4f6ead"
server_key_iv: "1e942c8e8bf1fb9ef19c3d5fd8f68bfa"
server_key_version: "encryptionkey@0"
I20260926 07:23:55.583272 18622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:55.587940 18622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:55.591241 18622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:55.591480 18622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:55.598680 18622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37243
I20260926 07:23:55.598707 18689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37243 every 8 connection(s)
I20260926 07:23:55.599622 18622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:55.602370 18690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:55.602988 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 18622
I20260926 07:23:55.603137 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:55.603412 29615 external_mini_cluster.cc:1428] Setting key 609cf64044eb98ece907a83037654487
I20260926 07:23:55.608675 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:55.604835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:51605 (local address 127.0.0.1:37243)
0926 07:23:55.605306 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:55.605321 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:55.605356 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:23:55.606073 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:55.606092 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:55.606384 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:55.606715 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:55.607539 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:55.607544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:55.607556 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:55.607567 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:55.607618 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:55.607636 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:55.607783 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:55.607852 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 07:23:55.608009 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":159,"threads_started":1}
I20260926 07:23:55.608852 18690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2: Bootstrap starting.
I20260926 07:23:55.611435 18690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:55.611583 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:55.612206 18690 log.cc:824] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:55.614372 18690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2: No bootstrap required, opened a new log
I20260926 07:23:55.615477 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:55.613734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:57667 (local address 127.0.0.1:37243)
0926 07:23:55.613869 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:55.613874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:55.614084 (+ 210us) server_negotiation.cc:407] Connection header received
0926 07:23:55.614228 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:55.614231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:55.614268 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:55.614351 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:55.615100 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:55.615104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:55.615107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:55.615118 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:55.615150 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:55.615161 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:55.615246 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:55.615317 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 07:23:55.615359 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:23:55.618652 18690 raft_consensus.cc:348] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38558bad86040d99cd6831d1315b2d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260926 07:23:55.618882 18690 raft_consensus.cc:374] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:55.618965 18690 raft_consensus.cc:729] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38558bad86040d99cd6831d1315b2d2, State: Initialized, Role: FOLLOWER
I20260926 07:23:55.619401 18690 consensus_queue.cc:260] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38558bad86040d99cd6831d1315b2d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260926 07:23:55.619547 18690 raft_consensus.cc:388] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:55.619637 18690 raft_consensus.cc:482] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:55.619751 18690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:55.620723 18690 raft_consensus.cc:504] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38558bad86040d99cd6831d1315b2d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260926 07:23:55.621132 18690 leader_election.cc:302] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f38558bad86040d99cd6831d1315b2d2; no voters:
I20260926 07:23:55.621487 18690 leader_election.cc:288] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:55.621636 18700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:55.621871 18700 raft_consensus.cc:686] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [term 1 LEADER]: Becoming Leader. State: Replica: f38558bad86040d99cd6831d1315b2d2, State: Running, Role: LEADER
I20260926 07:23:55.622239 18700 consensus_queue.cc:237] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38558bad86040d99cd6831d1315b2d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } }
I20260926 07:23:55.622776 18690 sys_catalog.cc:564] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:55.623961 18701 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f38558bad86040d99cd6831d1315b2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38558bad86040d99cd6831d1315b2d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } } }
I20260926 07:23:55.623958 18702 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f38558bad86040d99cd6831d1315b2d2. Latest consensus state: current_term: 1 leader_uuid: "f38558bad86040d99cd6831d1315b2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38558bad86040d99cd6831d1315b2d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37243 } } }
I20260926 07:23:55.624115 18701 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:55.624115 18702 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2 [sys.catalog]: This master's current role is: LEADER
W20260926 07:23:55.627599 18715 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:23:55.627677 18715 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:23:55.627763 18709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:55.628453 18709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:55.634151 18709 catalog_manager.cc:1347] Generated new cluster ID: 8f523c668e294ad99127501a3d7aa6e5
I20260926 07:23:55.634223 18709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:55.648229 18709 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:55.648567 18709 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:55.652932 18709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f38558bad86040d99cd6831d1315b2d2: Generated new TSK 0
I20260926 07:23:55.653644 18709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:55.660012 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 18622
2026-09-26T07:23:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:55 dist-test-slave-7zlw krb5kdc[18721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:55 dist-test-slave-7zlw krb5kdc[18721](info): set up 2 sockets
Sep 26 07:23:55 dist-test-slave-7zlw krb5kdc[18721](info): commencing operation
krb5kdc: starting...
W20260926 07:23:56.705386 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:56 dist-test-slave-7zlw krb5kdc[18721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:23:56.833590 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:36811
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:42447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:36811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:56.945798 18734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:56.946079 18734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:56.946164 18734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:56.950434 18734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:56.950502 18734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:56.950521 18734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:56.950538 18734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:56.950585 18734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:56.956002 18734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:42447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:36811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:36811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:56.957149 18734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:56.958225 18734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:23:56.966877 18734 server_base.cc:1034] running on GCE node
W20260926 07:23:56.966815 18742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:56.966843 18740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:56.966842 18739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:56.967680 18734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:56.968828 18734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:56.970024 18734 hybrid_clock.cc:648] HybridClock initialized: now 1790407436970009 us; error 34 us; skew 500 ppm
Sep 26 07:23:56 dist-test-slave-7zlw krb5kdc[18721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407436, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:56.972883 18734 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:23:56.973793 18734 webserver.cc:460] Webserver started at http://127.28.235.254:40203/ using document root <none> and password file <none>
I20260926 07:23:56.974366 18734 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:56.974426 18734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:56.974668 18734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:56.976821 18734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "619338e1d6ed44d78cc09de6e7bd9c64"
format_stamp: "Formatted at 2026-09-26 07:23:56 on dist-test-slave-7zlw"
server_key: "b111e85e4ca2c1508dc0afe83c2aaac0"
server_key_iv: "5b32547eef06b7269abb3f999db9227f"
server_key_version: "encryptionkey@0"
I20260926 07:23:56.977476 18734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "619338e1d6ed44d78cc09de6e7bd9c64"
format_stamp: "Formatted at 2026-09-26 07:23:56 on dist-test-slave-7zlw"
server_key: "b111e85e4ca2c1508dc0afe83c2aaac0"
server_key_iv: "5b32547eef06b7269abb3f999db9227f"
server_key_version: "encryptionkey@0"
I20260926 07:23:56.981494 18734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:23:56.983989 18749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:56.985440 18734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:56.985602 18734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "619338e1d6ed44d78cc09de6e7bd9c64"
format_stamp: "Formatted at 2026-09-26 07:23:56 on dist-test-slave-7zlw"
server_key: "b111e85e4ca2c1508dc0afe83c2aaac0"
server_key_iv: "5b32547eef06b7269abb3f999db9227f"
server_key_version: "encryptionkey@0"
I20260926 07:23:56.985707 18734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:57.005824 18734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:57.009192 18734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:57.009409 18734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:57.016927 18734 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:36811
I20260926 07:23:57.016949 18801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:36811 every 8 connection(s)
I20260926 07:23:57.017943 18734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:57.020067 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 18734
I20260926 07:23:57.020272 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:57.020576 29615 external_mini_cluster.cc:1428] Setting key 9b3bc2746688eb7aa7ea85c2160080ea
I20260926 07:23:57.021140 18802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:57.027225 18802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64: Bootstrap starting.
Sep 26 07:23:57 dist-test-slave-7zlw krb5kdc[18721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:23:57.029873 18802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:57.030643 18802 log.cc:824] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:57.032747 18802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64: No bootstrap required, opened a new log
I20260926 07:23:57.034967 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:57.022011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42535 (local address 127.28.235.254:36811)
0926 07:23:57.022456 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:57.022471 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:57.022507 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:23:57.023185 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:57.023202 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:57.023534 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:57.023880 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:57.024667 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:57.025812 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:57.026669 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:57.027150 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:57.027312 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:57.030253 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:57.030258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:57.030278 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:57.030306 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:57.032060 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:57.032579 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:57.032584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:57.032588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:57.032696 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:57.033095 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:57.033097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:57.033098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:57.033493 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:57.033709 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:57.034034 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 07:23:57.034252 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":148,"threads_started":1}
I20260926 07:23:57.036857 18802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619338e1d6ed44d78cc09de6e7bd9c64" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36811 } }
I20260926 07:23:57.037165 18802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:57.037249 18802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619338e1d6ed44d78cc09de6e7bd9c64, State: Initialized, Role: FOLLOWER
I20260926 07:23:57.037698 18802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619338e1d6ed44d78cc09de6e7bd9c64" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36811 } }
I20260926 07:23:57.037855 18802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:57.037925 18802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:57.038019 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:57.038041 18802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:57.038973 18802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619338e1d6ed44d78cc09de6e7bd9c64" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36811 } }
I20260926 07:23:57.039359 18802 leader_election.cc:302] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 619338e1d6ed44d78cc09de6e7bd9c64; no voters:
I20260926 07:23:57.039685 18802 leader_election.cc:288] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:57.039748 18811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:57.039983 18811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [term 1 LEADER]: Becoming Leader. State: Replica: 619338e1d6ed44d78cc09de6e7bd9c64, State: Running, Role: LEADER
I20260926 07:23:57.040338 18811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619338e1d6ed44d78cc09de6e7bd9c64" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36811 } }
I20260926 07:23:57.041008 18802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:57.041538 18811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "619338e1d6ed44d78cc09de6e7bd9c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619338e1d6ed44d78cc09de6e7bd9c64" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36811 } } }
I20260926 07:23:57.041663 18811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:57.041914 18811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 619338e1d6ed44d78cc09de6e7bd9c64. Latest consensus state: current_term: 1 leader_uuid: "619338e1d6ed44d78cc09de6e7bd9c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619338e1d6ed44d78cc09de6e7bd9c64" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 36811 } } }
I20260926 07:23:57.042013 18811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:57.042382 18818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:57.046231 18818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:57.053421 18818 catalog_manager.cc:1347] Generated new cluster ID: 29588449ee9c4aca8d2fdca505112f80
I20260926 07:23:57.053489 18818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:57.054049 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:57.039986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56003 (local address 127.28.235.254:36811)
0926 07:23:57.040101 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:57.040105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:57.043428 (+ 3323us) server_negotiation.cc:407] Connection header received
0926 07:23:57.043555 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:57.043558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:57.043605 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:57.043691 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:57.044503 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:57.045293 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:57.046411 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:57.046759 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:57.046836 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:57.048753 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:57.048757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:57.048760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:23:57.048783 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:57.051035 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:57.051972 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:57.051976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:57.051978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:57.052135 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:23:57.052738 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:23:57.052742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:23:57.052743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:23:57.053084 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:57.053262 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:57.053793 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 07:23:57.053899 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:23:57.060015 18818 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:57.061290 18818 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:57.068490 18818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 619338e1d6ed44d78cc09de6e7bd9c64: Generated new TSK 0
I20260926 07:23:57.069334 18818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:57.078668 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 18734
W20260926 07:23:57.084415 18719 connection.cc:535] client connection to 127.28.235.254:36811 recv error: Network error: failed to read from TLS socket (remote: 127.28.235.254:36811): Connection reset by peer (error 104)
2026-09-26T07:23:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:57 dist-test-slave-7zlw krb5kdc[18833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:57 dist-test-slave-7zlw krb5kdc[18833](info): set up 2 sockets
Sep 26 07:23:57 dist-test-slave-7zlw krb5kdc[18833](info): commencing operation
krb5kdc: starting...
W20260926 07:23:58.129434 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:58 dist-test-slave-7zlw krb5kdc[18833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:23:58.256649 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:42291
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:42291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:58.369242 18846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:58.369562 18846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:58.369640 18846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:58.373906 18846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:58.373989 18846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:58.374022 18846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:58.374058 18846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:58.374091 18846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:58.379593 18846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:42291
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:42291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:58.380779 18846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:58.381887 18846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:58.390654 18852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:58.390671 18851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:58.390820 18846 server_base.cc:1034] running on GCE node
W20260926 07:23:58.390654 18854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:58.391613 18846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:58.392798 18846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:58.393973 18846 hybrid_clock.cc:648] HybridClock initialized: now 1790407438393933 us; error 56 us; skew 500 ppm
Sep 26 07:23:58 dist-test-slave-7zlw krb5kdc[18833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407438, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:58.396929 18846 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:23:58.397917 18846 webserver.cc:460] Webserver started at http://127.28.235.254:38595/ using document root <none> and password file <none>
I20260926 07:23:58.398490 18846 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:58.398578 18846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:58.398805 18846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:58.401067 18846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "12808125a1444704ba15d415c173d56a"
format_stamp: "Formatted at 2026-09-26 07:23:58 on dist-test-slave-7zlw"
server_key: "1c582442bbb47ad570d49596a6e910d9"
server_key_iv: "04690568a61e259d1e508e82266042e8"
server_key_version: "encryptionkey@0"
I20260926 07:23:58.401693 18846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "12808125a1444704ba15d415c173d56a"
format_stamp: "Formatted at 2026-09-26 07:23:58 on dist-test-slave-7zlw"
server_key: "1c582442bbb47ad570d49596a6e910d9"
server_key_iv: "04690568a61e259d1e508e82266042e8"
server_key_version: "encryptionkey@0"
I20260926 07:23:58.405721 18846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:23:58.408295 18861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:58.409732 18846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:23:58.409873 18846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "12808125a1444704ba15d415c173d56a"
format_stamp: "Formatted at 2026-09-26 07:23:58 on dist-test-slave-7zlw"
server_key: "1c582442bbb47ad570d49596a6e910d9"
server_key_iv: "04690568a61e259d1e508e82266042e8"
server_key_version: "encryptionkey@0"
I20260926 07:23:58.409993 18846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:58.431378 18846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:58.434947 18846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:58.435156 18846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:58.442499 18846 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:42291
I20260926 07:23:58.442521 18913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:42291 every 8 connection(s)
I20260926 07:23:58.443486 18846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:58.446316 18914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:58.452044 18914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a: Bootstrap starting.
I20260926 07:23:58.453791 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 18846
I20260926 07:23:58.453931 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:58.454232 29615 external_mini_cluster.cc:1428] Setting key 36720e68919e50ff5afebfbc8cc33af3
I20260926 07:23:58.454898 18914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:58.455856 18914 log.cc:824] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:58.457932 18914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a: No bootstrap required, opened a new log
I20260926 07:23:58.461654 18914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12808125a1444704ba15d415c173d56a" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42291 } }
I20260926 07:23:58.461936 18914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:58.462011 18914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12808125a1444704ba15d415c173d56a, State: Initialized, Role: FOLLOWER
I20260926 07:23:58.462500 18914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12808125a1444704ba15d415c173d56a" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42291 } }
I20260926 07:23:58.462651 18914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:58.462762 18914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:58.462878 18914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:58.463675 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:58.455418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45397 (local address 127.28.235.254:42291)
0926 07:23:58.455870 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:58.455885 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:58.455919 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:23:58.456810 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:58.456829 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:58.457233 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:58.457560 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:58.457987 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:58.459044 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:58.459946 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:58.460453 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:58.460631 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:58.461614 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:58.461620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:58.461634 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:58.462192 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:58.462273 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:58.462299 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:58.462478 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:58.462673 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 07:23:58.462842 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":152,"threads_started":1}
I20260926 07:23:58.463964 18914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12808125a1444704ba15d415c173d56a" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42291 } }
I20260926 07:23:58.464309 18914 leader_election.cc:302] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12808125a1444704ba15d415c173d56a; no voters:
I20260926 07:23:58.464614 18914 leader_election.cc:288] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:58.464725 18919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:58.464969 18919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [term 1 LEADER]: Becoming Leader. State: Replica: 12808125a1444704ba15d415c173d56a, State: Running, Role: LEADER
I20260926 07:23:58.465358 18919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12808125a1444704ba15d415c173d56a" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42291 } }
I20260926 07:23:58.466069 18914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:58.467037 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:58.467027 18920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12808125a1444704ba15d415c173d56a. Latest consensus state: current_term: 1 leader_uuid: "12808125a1444704ba15d415c173d56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12808125a1444704ba15d415c173d56a" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42291 } } }
I20260926 07:23:58.467150 18920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:58.467344 18920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12808125a1444704ba15d415c173d56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12808125a1444704ba15d415c173d56a" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 42291 } } }
I20260926 07:23:58.467401 18920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:58.467686 18928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:58.471242 18928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:58.476609 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:58.469271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34813 (local address 127.28.235.254:42291)
0926 07:23:58.469385 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:58.469393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:58.469716 (+ 323us) server_negotiation.cc:407] Connection header received
0926 07:23:58.469862 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:58.469865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:58.469926 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:58.470008 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:58.470354 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:58.471088 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:58.472350 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:23:58.472668 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:23:58.472773 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:23:58.475990 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:58.475994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:58.475997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:58.476013 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:58.476048 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:58.476067 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:58.476161 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:58.476401 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 07:23:58.476439 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 07:23:58.478142 18928 catalog_manager.cc:1347] Generated new cluster ID: 7adf37ff201c49fa82e80da6f2f66b53
I20260926 07:23:58.478212 18928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:58.491442 18928 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:58.492421 18928 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:58.508153 18928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12808125a1444704ba15d415c173d56a: Generated new TSK 0
I20260926 07:23:58.508976 18928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:58.515034 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 18846
W20260926 07:23:58.520244 18831 connection.cc:535] client connection to 127.28.235.254:42291 recv error: Network error: failed to read from TLS socket (remote: 127.28.235.254:42291): Connection reset by peer (error 104)
2026-09-26T07:23:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:58 dist-test-slave-7zlw krb5kdc[18945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:23:58 dist-test-slave-7zlw krb5kdc[18945](info): set up 2 sockets
Sep 26 07:23:58 dist-test-slave-7zlw krb5kdc[18945](info): commencing operation
krb5kdc: starting...
W20260926 07:23:59.558084 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:23:59 dist-test-slave-7zlw krb5kdc[18945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:23:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:23:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:23:59.693149 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:38737
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:38737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:23:59.809278 18958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:23:59.809607 18958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:23:59.809684 18958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:23:59.813825 18958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:23:59.813928 18958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:23:59.813964 18958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:23:59.814011 18958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:23:59.814044 18958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:23:59.819715 18958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:38737
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:38737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:23:59.820887 18958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:23:59.821995 18958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:23:59.831609 18963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:23:59.831605 18966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:59.831831 18958 server_base.cc:1034] running on GCE node
W20260926 07:23:59.831605 18964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:23:59.832861 18958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:23:59.834244 18958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:23:59.835445 18958 hybrid_clock.cc:648] HybridClock initialized: now 1790407439835425 us; error 38 us; skew 500 ppm
Sep 26 07:23:59 dist-test-slave-7zlw krb5kdc[18945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407439, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:23:59.847628 18958 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:23:59.848765 18958 webserver.cc:460] Webserver started at http://127.28.235.254:45195/ using document root <none> and password file <none>
I20260926 07:23:59.849766 18958 fs_manager.cc:359] Metadata directory not provided
I20260926 07:23:59.849871 18958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:23:59.850117 18958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:23:59.852494 18958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "46cd5b1a4b1f41868133b68a13585b48"
format_stamp: "Formatted at 2026-09-26 07:23:59 on dist-test-slave-7zlw"
server_key: "2ece2b365f786455bfc26e6634fde961"
server_key_iv: "3bf31b5406ed8ca82a1f6616c4ad1946"
server_key_version: "encryptionkey@0"
I20260926 07:23:59.853194 18958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "46cd5b1a4b1f41868133b68a13585b48"
format_stamp: "Formatted at 2026-09-26 07:23:59 on dist-test-slave-7zlw"
server_key: "2ece2b365f786455bfc26e6634fde961"
server_key_iv: "3bf31b5406ed8ca82a1f6616c4ad1946"
server_key_version: "encryptionkey@0"
I20260926 07:23:59.857002 18958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:23:59.859670 18973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:23:59.860935 18958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:23:59.861104 18958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "46cd5b1a4b1f41868133b68a13585b48"
format_stamp: "Formatted at 2026-09-26 07:23:59 on dist-test-slave-7zlw"
server_key: "2ece2b365f786455bfc26e6634fde961"
server_key_iv: "3bf31b5406ed8ca82a1f6616c4ad1946"
server_key_version: "encryptionkey@0"
I20260926 07:23:59.861274 18958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:23:59.866066 18958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:23:59.869422 18958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:23:59.869638 18958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:23:59.877121 18958 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:38737
I20260926 07:23:59.877135 19025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:38737 every 8 connection(s)
I20260926 07:23:59.878351 18958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:23:59.880270 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 18958
I20260926 07:23:59.880419 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:23:59.880697 29615 external_mini_cluster.cc:1428] Setting key 04e4011c75524e7f95e8444c1ed7c34b
I20260926 07:23:59.881446 19026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:23:59.886523 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:59.882164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58173 (local address 127.28.235.254:38737)
0926 07:23:59.882593 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:59.882613 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:59.882666 (+ 53us) server_negotiation.cc:407] Connection header received
0926 07:23:59.883538 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:59.883562 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:59.883933 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:59.884306 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:59.885112 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:59.885118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:59.885133 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:59.885147 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:59.885213 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:59.885242 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:59.885389 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:59.885457 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 07:23:59.885608 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":114,"threads_started":1}
I20260926 07:23:59.888621 19026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48: Bootstrap starting.
I20260926 07:23:59.889464 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:23:59.891081 19026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48: Neither blocks nor log segments found. Creating new log.
I20260926 07:23:59.891958 19026 log.cc:824] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48: Log is configured to *not* fsync() on all Append() calls
I20260926 07:23:59.893554 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:23:59.891592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39205 (local address 127.28.235.254:38737)
0926 07:23:59.891736 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:23:59.891741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:23:59.891929 (+ 188us) server_negotiation.cc:407] Connection header received
0926 07:23:59.892100 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:23:59.892103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:23:59.892145 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:23:59.892215 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:23:59.893087 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:23:59.893094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:23:59.893097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:23:59.893148 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:23:59.893173 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:23:59.893184 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:23:59.893263 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:23:59.893366 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 07:23:59.893401 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:23:59.894316 19026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48: No bootstrap required, opened a new log
I20260926 07:23:59.897913 19026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46cd5b1a4b1f41868133b68a13585b48" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38737 } }
I20260926 07:23:59.898185 19026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:23:59.898247 19026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46cd5b1a4b1f41868133b68a13585b48, State: Initialized, Role: FOLLOWER
I20260926 07:23:59.898698 19026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46cd5b1a4b1f41868133b68a13585b48" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38737 } }
I20260926 07:23:59.898823 19026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:23:59.898901 19026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:23:59.899010 19026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:23:59.899942 19026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46cd5b1a4b1f41868133b68a13585b48" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38737 } }
I20260926 07:23:59.900290 19026 leader_election.cc:302] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46cd5b1a4b1f41868133b68a13585b48; no voters:
I20260926 07:23:59.900596 19026 leader_election.cc:288] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:23:59.900748 19036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:23:59.900991 19036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [term 1 LEADER]: Becoming Leader. State: Replica: 46cd5b1a4b1f41868133b68a13585b48, State: Running, Role: LEADER
I20260926 07:23:59.901337 19036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46cd5b1a4b1f41868133b68a13585b48" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38737 } }
I20260926 07:23:59.901839 19026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:23:59.902981 19038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46cd5b1a4b1f41868133b68a13585b48. Latest consensus state: current_term: 1 leader_uuid: "46cd5b1a4b1f41868133b68a13585b48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46cd5b1a4b1f41868133b68a13585b48" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38737 } } }
I20260926 07:23:59.903115 19038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:59.903425 19037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46cd5b1a4b1f41868133b68a13585b48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46cd5b1a4b1f41868133b68a13585b48" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 38737 } } }
I20260926 07:23:59.903549 19037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48 [sys.catalog]: This master's current role is: LEADER
I20260926 07:23:59.904189 19045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:23:59.907735 19045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:23:59.913986 19045 catalog_manager.cc:1347] Generated new cluster ID: 1745224f9d924c72a0d30cef0c9dfe01
I20260926 07:23:59.914090 19045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:23:59.932180 19045 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:23:59.932610 19045 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:23:59.936895 19045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46cd5b1a4b1f41868133b68a13585b48: Generated new TSK 0
I20260926 07:23:59.937639 19045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:23:59.944022 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 18958
2026-09-26T07:23:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:23:59 dist-test-slave-7zlw krb5kdc[19057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:24:00 dist-test-slave-7zlw krb5kdc[19057](info): set up 2 sockets
Sep 26 07:24:00 dist-test-slave-7zlw krb5kdc[19057](info): commencing operation
krb5kdc: starting...
W20260926 07:24:01.003625 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:24:01 dist-test-slave-7zlw krb5kdc[19057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:24:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:24:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:24:01.141378 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:35277
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:35277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:01.255926 19070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:01.256275 19070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:01.256348 19070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:01.260869 19070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:01.260998 19070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:01.261078 19070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:01.261123 19070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:01.261159 19070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:01.266825 19070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:35277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:35277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:01.267997 19070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:01.269107 19070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:24:01.278012 19070 server_base.cc:1034] running on GCE node
W20260926 07:24:01.277956 19076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:01.277976 19075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:01.277956 19078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:01.278857 19070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:01.280161 19070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:01.281358 19070 hybrid_clock.cc:648] HybridClock initialized: now 1790407441281330 us; error 40 us; skew 500 ppm
Sep 26 07:24:01 dist-test-slave-7zlw krb5kdc[19057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407441, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:01.284420 19070 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:24:01.285503 19070 webserver.cc:460] Webserver started at http://127.28.235.254:39535/ using document root <none> and password file <none>
I20260926 07:24:01.286100 19070 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:01.286162 19070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:01.286386 19070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:01.288486 19070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "347e912f670a463bb5fe1e04cdd0d5e4"
format_stamp: "Formatted at 2026-09-26 07:24:01 on dist-test-slave-7zlw"
server_key: "695df9d65e6c5ff718cf3f42da91ae2c"
server_key_iv: "47420cce14f285f9742abc8592ba1db8"
server_key_version: "encryptionkey@0"
I20260926 07:24:01.289157 19070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "347e912f670a463bb5fe1e04cdd0d5e4"
format_stamp: "Formatted at 2026-09-26 07:24:01 on dist-test-slave-7zlw"
server_key: "695df9d65e6c5ff718cf3f42da91ae2c"
server_key_iv: "47420cce14f285f9742abc8592ba1db8"
server_key_version: "encryptionkey@0"
I20260926 07:24:01.293449 19070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:24:01.295960 19085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:01.297315 19070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:24:01.297425 19070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "347e912f670a463bb5fe1e04cdd0d5e4"
format_stamp: "Formatted at 2026-09-26 07:24:01 on dist-test-slave-7zlw"
server_key: "695df9d65e6c5ff718cf3f42da91ae2c"
server_key_iv: "47420cce14f285f9742abc8592ba1db8"
server_key_version: "encryptionkey@0"
I20260926 07:24:01.297561 19070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:01.326040 19070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:01.329658 19070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:01.329921 19070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:01.337641 19070 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:35277
I20260926 07:24:01.337670 19137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:35277 every 8 connection(s)
I20260926 07:24:01.338644 19070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:24:01.341610 19138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:01.347437 19138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4: Bootstrap starting.
I20260926 07:24:01.348740 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 19070
I20260926 07:24:01.348896 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:24:01.349409 29615 external_mini_cluster.cc:1428] Setting key 4377d3fc744675dd32e51568f0bb8406
I20260926 07:24:01.350782 19138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:01.351678 19138 log.cc:824] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:01.353843 19138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4: No bootstrap required, opened a new log
I20260926 07:24:01.357810 19138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e912f670a463bb5fe1e04cdd0d5e4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35277 } }
I20260926 07:24:01.358136 19138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:24:01.358197 19138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347e912f670a463bb5fe1e04cdd0d5e4, State: Initialized, Role: FOLLOWER
Sep 26 07:24:01 dist-test-slave-7zlw krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:24:01.358709 19138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e912f670a463bb5fe1e04cdd0d5e4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35277 } }
I20260926 07:24:01.358841 19138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:24:01.358927 19138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:24:01.359043 19138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:01.360272 19138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e912f670a463bb5fe1e04cdd0d5e4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35277 } }
I20260926 07:24:01.360662 19138 leader_election.cc:302] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 347e912f670a463bb5fe1e04cdd0d5e4; no voters:
I20260926 07:24:01.360994 19138 leader_election.cc:288] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:24:01.361179 19143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:24:01.361395 19143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [term 1 LEADER]: Becoming Leader. State: Replica: 347e912f670a463bb5fe1e04cdd0d5e4, State: Running, Role: LEADER
I20260926 07:24:01.361797 19143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e912f670a463bb5fe1e04cdd0d5e4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35277 } }
I20260926 07:24:01.362430 19138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:01.363744 19144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "347e912f670a463bb5fe1e04cdd0d5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e912f670a463bb5fe1e04cdd0d5e4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35277 } } }
I20260926 07:24:01.363878 19144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:01.364256 19145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 347e912f670a463bb5fe1e04cdd0d5e4. Latest consensus state: current_term: 1 leader_uuid: "347e912f670a463bb5fe1e04cdd0d5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347e912f670a463bb5fe1e04cdd0d5e4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 35277 } } }
I20260926 07:24:01.365219 19145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:01.365397 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:01.350756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54395 (local address 127.28.235.254:35277)
0926 07:24:01.351270 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:01.351287 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:01.351323 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:24:01.352079 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:01.352106 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:01.352566 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:01.352916 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:01.353748 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:01.354786 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:01.355648 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:01.356163 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:01.356348 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:01.359180 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:01.359187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:01.359250 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:01.359281 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:01.361406 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:01.361945 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:01.361951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:01.361957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:01.362092 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:01.362431 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:01.362434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:01.362436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:01.362976 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:01.363176 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:01.363983 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 07:24:01.364321 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":168,"threads_started":1}
I20260926 07:24:01.368142 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 07:24:01.369889 19158 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:24:01.369969 19158 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:24:01.370054 19152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:24:01.371131 19152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:24:01.378520 19152 catalog_manager.cc:1347] Generated new cluster ID: 74db451326f048d2a6d7c103f0e2ab93
I20260926 07:24:01.378595 19152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:24:01.379259 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:01.370614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40455 (local address 127.28.235.254:35277)
0926 07:24:01.370737 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:01.370742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:01.370964 (+ 222us) server_negotiation.cc:407] Connection header received
0926 07:24:01.371083 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:01.371086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:01.371139 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:01.371238 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:01.372258 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:01.372995 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:01.373867 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:01.374106 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:01.374224 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:01.375607 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:01.375611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:01.375613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:01.375643 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:01.377363 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:01.377906 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:01.377909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:01.377911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:01.378010 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:01.378356 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:01.378359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:01.378360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:01.378586 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:01.378728 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:01.378969 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 07:24:01.379088 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:24:01.391661 19152 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:24:01.392643 19152 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:24:01.402742 19152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 347e912f670a463bb5fe1e04cdd0d5e4: Generated new TSK 0
I20260926 07:24:01.403620 19152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:24:01.408036 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 19070
2026-09-26T07:24:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:24:01 dist-test-slave-7zlw krb5kdc[19169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:24:01 dist-test-slave-7zlw krb5kdc[19169](info): set up 2 sockets
Sep 26 07:24:01 dist-test-slave-7zlw krb5kdc[19169](info): commencing operation
krb5kdc: starting...
W20260926 07:24:02.473351 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:24:02 dist-test-slave-7zlw krb5kdc[19169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:24:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:24:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:24:02.608772 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:46075
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:46075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:02.724342 19182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:02.724710 19182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:02.724799 19182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:02.729069 19182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:02.729220 19182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:02.729264 19182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:02.729301 19182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:02.729336 19182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:02.735013 19182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:46075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:46075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:02.736383 19182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:02.737461 19182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:24:02.746656 19182 server_base.cc:1034] running on GCE node
W20260926 07:24:02.746592 19187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:02.746596 19190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:02.746595 19188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:02.747560 19182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:02.748752 19182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:02.749959 19182 hybrid_clock.cc:648] HybridClock initialized: now 1790407442749937 us; error 38 us; skew 500 ppm
Sep 26 07:24:02 dist-test-slave-7zlw krb5kdc[19169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407442, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:02.753131 19182 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:24:02.754122 19182 webserver.cc:460] Webserver started at http://127.28.235.254:43907/ using document root <none> and password file <none>
I20260926 07:24:02.754698 19182 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:02.754761 19182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:02.755016 19182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:02.757181 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "d7e834d79a8047908ecffd444ad5ca2e"
format_stamp: "Formatted at 2026-09-26 07:24:02 on dist-test-slave-7zlw"
server_key: "191ebbaa78f62f6fc399a10b846d25ac"
server_key_iv: "f564b38b68e197b5bd59c5ebcc73a406"
server_key_version: "encryptionkey@0"
I20260926 07:24:02.757761 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "d7e834d79a8047908ecffd444ad5ca2e"
format_stamp: "Formatted at 2026-09-26 07:24:02 on dist-test-slave-7zlw"
server_key: "191ebbaa78f62f6fc399a10b846d25ac"
server_key_iv: "f564b38b68e197b5bd59c5ebcc73a406"
server_key_version: "encryptionkey@0"
I20260926 07:24:02.761868 19182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 07:24:02.764408 19197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:02.766026 19182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:24:02.766206 19182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "d7e834d79a8047908ecffd444ad5ca2e"
format_stamp: "Formatted at 2026-09-26 07:24:02 on dist-test-slave-7zlw"
server_key: "191ebbaa78f62f6fc399a10b846d25ac"
server_key_iv: "f564b38b68e197b5bd59c5ebcc73a406"
server_key_version: "encryptionkey@0"
I20260926 07:24:02.766338 19182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:02.788515 19182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:02.792052 19182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:02.792305 19182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:02.800431 19182 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:46075
I20260926 07:24:02.800484 19249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:46075 every 8 connection(s)
I20260926 07:24:02.801620 19182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:24:02.804941 19250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:02.806007 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 19182
I20260926 07:24:02.806257 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:24:02.806569 29615 external_mini_cluster.cc:1428] Setting key 3334918052dc0545e9b38b21ae470f86
I20260926 07:24:02.811697 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e: Bootstrap starting.
I20260926 07:24:02.814728 19250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:02.815582 19250 log.cc:824] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:02.816597 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:02.808176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40897 (local address 127.28.235.254:46075)
0926 07:24:02.808656 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:02.808673 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:02.808712 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:24:02.809571 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:02.809591 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:02.809971 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:02.810274 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:02.810748 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:02.811846 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:02.813193 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:02.813666 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:02.813792 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:02.814696 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:02.814701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:02.814715 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:24:02.815211 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:02.815269 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:24:02.815293 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:02.815373 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:02.815561 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 07:24:02.815783 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":151,"threads_started":1}
I20260926 07:24:02.817785 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e: No bootstrap required, opened a new log
I20260926 07:24:02.819664 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:24:02.822391 19250 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e834d79a8047908ecffd444ad5ca2e" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46075 } }
I20260926 07:24:02.822736 19250 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:24:02.822819 19250 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7e834d79a8047908ecffd444ad5ca2e, State: Initialized, Role: FOLLOWER
I20260926 07:24:02.823266 19250 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e834d79a8047908ecffd444ad5ca2e" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46075 } }
I20260926 07:24:02.823392 19250 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:24:02.823480 19250 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:24:02.823603 19250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:02.824908 19250 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e834d79a8047908ecffd444ad5ca2e" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46075 } }
I20260926 07:24:02.825392 19250 leader_election.cc:302] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7e834d79a8047908ecffd444ad5ca2e; no voters:
I20260926 07:24:02.825803 19250 leader_election.cc:288] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:24:02.825937 19260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:24:02.826208 19260 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [term 1 LEADER]: Becoming Leader. State: Replica: d7e834d79a8047908ecffd444ad5ca2e, State: Running, Role: LEADER
I20260926 07:24:02.826624 19260 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e834d79a8047908ecffd444ad5ca2e" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46075 } }
I20260926 07:24:02.827020 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:02.821742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56863 (local address 127.28.235.254:46075)
0926 07:24:02.821876 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:02.821881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:02.822059 (+ 178us) server_negotiation.cc:407] Connection header received
0926 07:24:02.822190 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:02.822192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:02.822233 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:02.822298 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:02.822678 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:02.823567 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:02.824853 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:02.825358 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:02.825457 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:02.826467 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:02.826471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:02.826474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:24:02.826488 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:02.826518 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:24:02.826532 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:02.826579 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:02.826815 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 07:24:02.826879 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:24:02.827504 19250 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:02.828087 19262 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7e834d79a8047908ecffd444ad5ca2e. Latest consensus state: current_term: 1 leader_uuid: "d7e834d79a8047908ecffd444ad5ca2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e834d79a8047908ecffd444ad5ca2e" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46075 } } }
I20260926 07:24:02.828361 19262 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:02.828516 19261 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7e834d79a8047908ecffd444ad5ca2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e834d79a8047908ecffd444ad5ca2e" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 46075 } } }
I20260926 07:24:02.828629 19261 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:02.828845 19267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:24:02.832424 19267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:24:02.838553 19267 catalog_manager.cc:1347] Generated new cluster ID: c9cbd44218984b27ba5ba7e15a11a9b4
I20260926 07:24:02.838656 19267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:24:02.856976 19267 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:24:02.858168 19267 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:24:02.863293 19267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7e834d79a8047908ecffd444ad5ca2e: Generated new TSK 0
I20260926 07:24:02.864014 19267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:24:02.871733 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 19182
2026-09-26T07:24:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:24:02 dist-test-slave-7zlw krb5kdc[19281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:24:02 dist-test-slave-7zlw krb5kdc[19281](info): set up 2 sockets
Sep 26 07:24:02 dist-test-slave-7zlw krb5kdc[19281](info): commencing operation
krb5kdc: starting...
W20260926 07:24:03.921378 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:24:04 dist-test-slave-7zlw krb5kdc[19281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:24:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:24:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:24:04.054019 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:33911
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:33911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:04.168216 19294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:04.168525 19294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:04.168570 19294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:04.172828 19294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:04.172911 19294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:04.172931 19294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:04.172948 19294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:04.172964 19294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:04.178591 19294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:33911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:33911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:04.179750 19294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:04.180815 19294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:04.189468 19302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:04.189498 19299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:04.189502 19300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:04.189739 19294 server_base.cc:1034] running on GCE node
I20260926 07:24:04.190306 19294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:04.191494 19294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:04.192694 19294 hybrid_clock.cc:648] HybridClock initialized: now 1790407444192663 us; error 44 us; skew 500 ppm
Sep 26 07:24:04 dist-test-slave-7zlw krb5kdc[19281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407444, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:04.195736 19294 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:24:04.196731 19294 webserver.cc:460] Webserver started at http://127.28.235.254:46469/ using document root <none> and password file <none>
I20260926 07:24:04.197413 19294 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:04.197504 19294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:04.197733 19294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:04.199776 19294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3"
format_stamp: "Formatted at 2026-09-26 07:24:04 on dist-test-slave-7zlw"
server_key: "cae9d3627fb7119078071f8e35f308d1"
server_key_iv: "0123d48e1867e9f6a96c9f36f67e69bc"
server_key_version: "encryptionkey@0"
I20260926 07:24:04.200338 19294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3"
format_stamp: "Formatted at 2026-09-26 07:24:04 on dist-test-slave-7zlw"
server_key: "cae9d3627fb7119078071f8e35f308d1"
server_key_iv: "0123d48e1867e9f6a96c9f36f67e69bc"
server_key_version: "encryptionkey@0"
I20260926 07:24:04.204108 19294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 07:24:04.206701 19309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:04.208038 19294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:24:04.208218 19294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3"
format_stamp: "Formatted at 2026-09-26 07:24:04 on dist-test-slave-7zlw"
server_key: "cae9d3627fb7119078071f8e35f308d1"
server_key_iv: "0123d48e1867e9f6a96c9f36f67e69bc"
server_key_version: "encryptionkey@0"
I20260926 07:24:04.208341 19294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:04.213119 19294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:04.216434 19294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:04.216637 19294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:04.224021 19294 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:33911
I20260926 07:24:04.224051 19361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:33911 every 8 connection(s)
I20260926 07:24:04.225117 19294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:24:04.228020 19362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:04.230826 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 19294
I20260926 07:24:04.230971 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:24:04.231280 29615 external_mini_cluster.cc:1428] Setting key e0c3f948559d3bba522d35a41fd922fb
I20260926 07:24:04.234861 19362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3: Bootstrap starting.
I20260926 07:24:04.236881 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:04.232730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60851 (local address 127.28.235.254:33911)
0926 07:24:04.233197 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:04.233214 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:04.233260 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:24:04.233904 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:04.233920 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:04.234222 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:04.234506 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:04.235472 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:04.235477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:04.235489 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:24:04.235499 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:04.235555 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:24:04.235574 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:04.235691 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:04.235966 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:24:04.236152 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":145,"threads_started":1}
I20260926 07:24:04.237679 19362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:04.238493 19362 log.cc:824] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:04.240139 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:24:04.240860 19362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3: No bootstrap required, opened a new log
I20260926 07:24:04.244354 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:04.242515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54331 (local address 127.28.235.254:33911)
0926 07:24:04.242641 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:04.242645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:04.242839 (+ 194us) server_negotiation.cc:407] Connection header received
0926 07:24:04.242992 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:04.242994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:04.243027 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:04.243124 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:04.243842 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:04.243844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:04.243846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:24:04.243857 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:04.243884 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:24:04.243894 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:04.243969 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:04.244129 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 07:24:04.244164 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:24:04.245442 19362 raft_consensus.cc:348] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33911 } }
I20260926 07:24:04.245692 19362 raft_consensus.cc:374] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:24:04.245774 19362 raft_consensus.cc:729] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a961bef20bdb4e6cb4c6ae9aa18e06e3, State: Initialized, Role: FOLLOWER
I20260926 07:24:04.246212 19362 consensus_queue.cc:260] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33911 } }
I20260926 07:24:04.246340 19362 raft_consensus.cc:388] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:24:04.246403 19362 raft_consensus.cc:482] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:24:04.246520 19362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:04.247525 19362 raft_consensus.cc:504] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33911 } }
I20260926 07:24:04.247884 19362 leader_election.cc:302] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a961bef20bdb4e6cb4c6ae9aa18e06e3; no voters:
I20260926 07:24:04.248202 19362 leader_election.cc:288] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:24:04.248404 19372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:24:04.248723 19372 raft_consensus.cc:686] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [term 1 LEADER]: Becoming Leader. State: Replica: a961bef20bdb4e6cb4c6ae9aa18e06e3, State: Running, Role: LEADER
I20260926 07:24:04.249155 19372 consensus_queue.cc:237] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33911 } }
I20260926 07:24:04.249644 19362 sys_catalog.cc:564] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:04.251456 19373 sys_catalog.cc:455] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33911 } } }
I20260926 07:24:04.251623 19373 sys_catalog.cc:458] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:04.252161 19374 sys_catalog.cc:455] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a961bef20bdb4e6cb4c6ae9aa18e06e3. Latest consensus state: current_term: 1 leader_uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a961bef20bdb4e6cb4c6ae9aa18e06e3" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33911 } } }
I20260926 07:24:04.252269 19374 sys_catalog.cc:458] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3 [sys.catalog]: This master's current role is: LEADER
W20260926 07:24:04.254911 19386 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:24:04.254993 19386 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:24:04.255097 19387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:24:04.255956 19387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:24:04.261869 19387 catalog_manager.cc:1347] Generated new cluster ID: 27d1dcb2a93a498788cd274c541bb243
I20260926 07:24:04.261953 19387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:24:04.273859 19387 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:24:04.274294 19387 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:24:04.284391 19387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a961bef20bdb4e6cb4c6ae9aa18e06e3: Generated new TSK 0
I20260926 07:24:04.285290 19387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:24:04.301502 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 19294
2026-09-26T07:24:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1430 ms)
[----------] 18 tests from AuthTokenIssuingTest (26025 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:24:04 dist-test-slave-7zlw krb5kdc[19393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:24:04 dist-test-slave-7zlw krb5kdc[19393](info): set up 2 sockets
Sep 26 07:24:04 dist-test-slave-7zlw krb5kdc[19393](info): commencing operation
krb5kdc: starting...
W20260926 07:24:05.345996 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:24:05 dist-test-slave-7zlw krb5kdc[19393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:24:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:24:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:24:05.477682 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:33437
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:33437,127.28.235.253:36735,127.28.235.252:44885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:05.594117 19406 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:24:05.594733 19406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:05.594807 19406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:05.594887 19406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:05.599129 19406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:05.599243 19406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:05.599283 19406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:05.599323 19406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:05.599351 19406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:05.606133 19406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:33437,127.28.235.253:36735,127.28.235.252:44885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:33437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:05.607391 19406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:05.608464 19406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:05.617551 19411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:05.617551 19412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:05.617609 19414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:05.618040 19406 server_base.cc:1034] running on GCE node
I20260926 07:24:05.618707 19406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:05.620040 19406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:05.621166 19406 hybrid_clock.cc:648] HybridClock initialized: now 1790407445621152 us; error 33 us; skew 500 ppm
Sep 26 07:24:05 dist-test-slave-7zlw krb5kdc[19393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:05.629388 19406 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:24:05.630524 19406 webserver.cc:460] Webserver started at http://127.28.235.254:42827/ using document root <none> and password file <none>
I20260926 07:24:05.631157 19406 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:05.631258 19406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:05.631510 19406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:05.633919 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "eda7cd6c48e641ffa3c8a72bf880aca4"
format_stamp: "Formatted at 2026-09-26 07:24:05 on dist-test-slave-7zlw"
server_key: "1059463421032e13357c512eef57797a"
server_key_iv: "bd0ff4b13009cd8eb1dc27bf7b29226f"
server_key_version: "encryptionkey@0"
I20260926 07:24:05.634542 19406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "eda7cd6c48e641ffa3c8a72bf880aca4"
format_stamp: "Formatted at 2026-09-26 07:24:05 on dist-test-slave-7zlw"
server_key: "1059463421032e13357c512eef57797a"
server_key_iv: "bd0ff4b13009cd8eb1dc27bf7b29226f"
server_key_version: "encryptionkey@0"
I20260926 07:24:05.638613 19406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:24:05.641377 19421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:05.642647 19406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:24:05.642810 19406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "eda7cd6c48e641ffa3c8a72bf880aca4"
format_stamp: "Formatted at 2026-09-26 07:24:05 on dist-test-slave-7zlw"
server_key: "1059463421032e13357c512eef57797a"
server_key_iv: "bd0ff4b13009cd8eb1dc27bf7b29226f"
server_key_version: "encryptionkey@0"
I20260926 07:24:05.642940 19406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:05.662349 19406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:05.665928 19406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:05.666172 19406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:05.674093 19406 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:33437
I20260926 07:24:05.674415 19473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:33437 every 8 connection(s)
I20260926 07:24:05.675168 19406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:24:05.678220 19474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:05.682068 19474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:24:05.685082 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 19406
I20260926 07:24:05.685216 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:24:05.685528 29615 external_mini_cluster.cc:1428] Setting key 3a736c1e0b2904391f567b04c57d5350
Sep 26 07:24:05 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.235.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.253@KRBTEST.COM" created.
I20260926 07:24:05.699934 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.684674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:33437 (local address 127.0.0.1:54342)
0926 07:24:05.685393 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.685452 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:05.686785 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:05.687722 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:05.687727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:05.688127 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:05.688379 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:05.688391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.689705 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.689712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:05.691178 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:05.691194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.691921 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.691928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:05.692055 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:05.692816 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:05.692839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:05.694596 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:05.696626 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:05.696635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:05.696647 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:05.696871 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:05.697268 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:05.697271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:05.697273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:05.697387 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:05.698154 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:05.698161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:05.698306 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 07:24:05.698476 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 07:24:05.698887 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":116,"thread_start_us":168,"threads_started":1}
I20260926 07:24:05.699934 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.684674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.28.235.254:33437)
0926 07:24:05.685329 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:05.685348 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:05.685587 (+ 239us) server_negotiation.cc:407] Connection header received
0926 07:24:05.687207 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:05.687215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:05.687485 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:05.687705 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:05.688540 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.689541 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.691341 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.691728 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.691852 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:05.694761 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:05.694766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:05.694777 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:05.694799 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:05.696470 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:05.696982 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:05.696988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:05.696990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:05.697165 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:05.697482 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:05.697485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:05.697487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:05.698025 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:05.698177 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:05.698549 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 07:24:05.698833 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":511,"spinlock_wait_cycles":55168,"thread_start_us":168,"threads_started":1}
I20260926 07:24:05.702651 19474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:24:05.703500 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.703237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51526)
0926 07:24:05.703364 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.703435 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:24:05.703689 19423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
I20260926 07:24:05.704582 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.704372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51540)
0926 07:24:05.704480 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.704531 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:24:05.704742 19474 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.253:36735: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab.
Entry for principal kudu/127.28.235.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab.
I20260926 07:24:05.717664 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.253:36735
--webserver_interface=127.28.235.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab
--principal=kudu/127.28.235.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:33437,127.28.235.253:36735,127.28.235.252:44885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:24:05.754011 19474 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } attempt: 1
I20260926 07:24:05.755031 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.754620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51552)
0926 07:24:05.754819 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.754900 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 07:24:05.755843 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.755606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51564)
0926 07:24:05.755724 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.755780 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:24:05.756031 19474 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.253:36735: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
I20260926 07:24:05.824255 19474 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } attempt: 2
I20260926 07:24:05.825187 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.824817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51566)
0926 07:24:05.824993 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.825091 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 07:24:05.825994 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.825763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51572)
0926 07:24:05.825873 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.825928 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:24:05.826130 19474 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.253:36735: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
W20260926 07:24:05.835680 19481 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:24:05.836009 19481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:05.836040 19481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:05.836081 19481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:05.840250 19481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:05.840324 19481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:05.840344 19481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:05.840363 19481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:05.840377 19481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:05.846079 19481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:33437,127.28.235.253:36735,127.28.235.252:44885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab
--principal=kudu/127.28.235.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.253:36735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.235.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:05.847234 19481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:05.848307 19481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:05.856917 19487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:05.856922 19486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:05.856966 19481 server_base.cc:1034] running on GCE node
W20260926 07:24:05.856946 19489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:05.857834 19481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:05.859055 19481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:05.860270 19481 hybrid_clock.cc:648] HybridClock initialized: now 1790407445860249 us; error 33 us; skew 500 ppm
Sep 26 07:24:05 dist-test-slave-7zlw krb5kdc[19393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:05.863374 19481 init.cc:377] Logged in from keytab as kudu/127.28.235.253@KRBTEST.COM (short username kudu)
I20260926 07:24:05.864400 19481 webserver.cc:460] Webserver started at http://127.28.235.253:40771/ using document root <none> and password file <none>
I20260926 07:24:05.865072 19481 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:05.865163 19481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:05.865377 19481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:05.867540 19481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/data/instance:
uuid: "eeb3a1fdf95248728691c39445c09ed3"
format_stamp: "Formatted at 2026-09-26 07:24:05 on dist-test-slave-7zlw"
server_key: "859a3a7aa37595d37dbfe7f73e8c307b"
server_key_iv: "d23581e60646edc48d1dd5d7bd00d1ef"
server_key_version: "encryptionkey@0"
I20260926 07:24:05.868167 19481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/wal/instance:
uuid: "eeb3a1fdf95248728691c39445c09ed3"
format_stamp: "Formatted at 2026-09-26 07:24:05 on dist-test-slave-7zlw"
server_key: "859a3a7aa37595d37dbfe7f73e8c307b"
server_key_iv: "d23581e60646edc48d1dd5d7bd00d1ef"
server_key_version: "encryptionkey@0"
I20260926 07:24:05.872236 19481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:24:05.875021 19496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:05.876402 19481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 07:24:05.876595 19481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/wal
uuid: "eeb3a1fdf95248728691c39445c09ed3"
format_stamp: "Formatted at 2026-09-26 07:24:05 on dist-test-slave-7zlw"
server_key: "859a3a7aa37595d37dbfe7f73e8c307b"
server_key_iv: "d23581e60646edc48d1dd5d7bd00d1ef"
server_key_version: "encryptionkey@0"
I20260926 07:24:05.876729 19481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:05.911893 19481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:05.915223 19481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:05.915447 19481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:05.917306 19474 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } attempt: 3
I20260926 07:24:05.918242 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.917877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51580)
0926 07:24:05.918062 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.918137 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 07:24:05.919016 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.918790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51582)
0926 07:24:05.918907 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.918957 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:24:05.919255 19474 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.253:36735: Network error: Client connection negotiation failed: client connection to 127.28.235.253:36735: connect: Connection refused (error 111)
I20260926 07:24:05.923439 19481 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.253:36735
I20260926 07:24:05.923765 19548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.253:36735 every 8 connection(s)
I20260926 07:24:05.924528 19481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
I20260926 07:24:05.925292 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 19481
I20260926 07:24:05.925406 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-1/wal/instance
I20260926 07:24:05.925714 29615 external_mini_cluster.cc:1428] Setting key afb01050895fbff95795cddd14a61a51
I20260926 07:24:05.930282 19549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
WARNING: no policy specified for kudu/127.28.235.252@KRBTEST.COM; defaulting to no policy
I20260926 07:24:05.935541 19549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.252@KRBTEST.COM" created.
Sep 26 07:24:05 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:24:05.951695 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.937698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.28.235.254:33437)
0926 07:24:05.937845 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:05.937849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:05.938889 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 07:24:05.939925 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:05.939929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:05.939992 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:05.940105 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:05.941521 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.942164 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.944143 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.944403 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.944558 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:05.947625 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:05.947629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:05.947631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:05.947657 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:05.949370 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:05.949884 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:05.949887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:05.949888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:05.949962 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:05.950358 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:05.950361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:05.950362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:05.950580 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:05.950718 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:05.951406 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 07:24:05.951560 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:24:05.952800 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.938134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:33437 (local address 127.0.0.1:54358)
0926 07:24:05.938708 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.938761 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:05.939667 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:05.940280 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:05.940288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:05.940651 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:05.941328 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:05.941351 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.942318 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.942325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:05.943932 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:05.943948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.944601 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.944605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:05.944842 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:05.945544 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:05.945568 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:05.947403 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:05.949493 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:05.949500 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:05.949524 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:05.949790 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:05.950102 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:05.950104 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:05.950106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:05.950245 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:05.950735 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:05.950741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:05.951104 (+ 363us) client_negotiation.cc:769] Sending connection context
0926 07:24:05.951290 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 07:24:05.951724 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":147,"threads_started":1}
I20260926 07:24:05.954609 19549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab.
Entry for principal kudu/127.28.235.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab.
I20260926 07:24:05.957476 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.252:44885
--webserver_interface=127.28.235.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab
--principal=kudu/127.28.235.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:36601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:33437,127.28.235.253:36735,127.28.235.252:44885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:24:05 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
I20260926 07:24:05.966212 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.955196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51584)
0926 07:24:05.955354 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.955368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:05.955441 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:05.956116 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:05.956119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:05.956392 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:05.956530 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:05.956541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.958121 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.958124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:05.958692 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:05.958701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.959341 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.959344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:05.959440 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:05.960020 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:05.960041 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:05.961703 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:05.964327 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:05.964331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:05.964332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:05.964564 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:05.964940 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:05.964942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:05.964944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:05.965105 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:05.965747 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:05.965749 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:05.965836 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 07:24:05.965929 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:24:05.966051 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 07:24:05.966518 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.955354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.28.235.253:36735)
0926 07:24:05.955694 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:05.955705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:05.955725 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:24:05.955762 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:05.955767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:05.955920 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:05.956029 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:05.956643 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.957879 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.958838 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:05.959220 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:05.959341 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:05.961866 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:05.961882 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:05.961906 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:05.961940 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:05.964179 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:05.964713 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:05.964722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:05.964726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:05.964835 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:05.965209 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:05.965216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:05.965219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:05.965623 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:05.965790 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:05.966066 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 07:24:05.966260 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":137,"threads_started":1}
I20260926 07:24:05.968881 19549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:24:05.969771 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.969452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50610)
0926 07:24:05.969589 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.969703 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:24:05.970017 19499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
I20260926 07:24:05.970907 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:05.970687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50624)
0926 07:24:05.970801 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:05.970855 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:24:05.971083 19549 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:44885: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
I20260926 07:24:06.020350 19549 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } attempt: 1
I20260926 07:24:06.021409 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.020985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50628)
0926 07:24:06.021221 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.021326 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 07:24:06.022152 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.021926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50634)
0926 07:24:06.022038 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.022096 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:24:06.022293 19549 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:44885: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
I20260926 07:24:06.062474 19474 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } attempt: 4
Sep 26 07:24:06 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
I20260926 07:24:06.073185 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.063099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51600)
0926 07:24:06.063406 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.063424 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:06.063542 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:06.063834 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:06.063837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:06.064045 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:06.064633 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.064644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.065870 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.065874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.066766 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.066777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.067324 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.067328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.067433 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.067961 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:06.067976 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:06.069533 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:06.071417 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.071422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.071428 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.071653 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.072019 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.072023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.072025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.072128 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.072586 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:06.072589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:06.072684 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 07:24:06.072794 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:24:06.072927 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260926 07:24:06.073202 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.063260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.28.235.253:36735)
0926 07:24:06.063409 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.063415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.063508 (+ 93us) server_negotiation.cc:407] Connection header received
0926 07:24:06.063640 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.063642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.063709 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.063783 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.064774 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.065743 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.066924 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.067188 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.067287 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.069692 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.069696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.069698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.069724 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.071292 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.071792 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.071796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.071798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.071894 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.072243 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.072246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.072248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.072469 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.072568 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.072865 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.072996 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:24:06.074120 19474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:24:06.074899 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.074590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50636)
0926 07:24:06.074754 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.074823 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:24:06.075667 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.075456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50640)
0926 07:24:06.075562 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.075613 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 07:24:06.075824 19474 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:44885: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
W20260926 07:24:06.077812 19555 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:24:06.078096 19555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:06.078120 19555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:06.078163 19555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:06.082489 19555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:06.082546 19555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:06.082566 19555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:06.082617 19555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:06.082650 19555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:06.089450 19555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:36601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:33437,127.28.235.253:36735,127.28.235.252:44885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab
--principal=kudu/127.28.235.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.252:44885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.235.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:06.090443 19549 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } attempt: 2
I20260926 07:24:06.090667 19555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:06.091265 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.090958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50648)
0926 07:24:06.091099 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.091183 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:24:06.091780 19555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:24:06.091986 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.091770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50650)
0926 07:24:06.091876 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.091924 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 07:24:06.092130 19549 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:44885: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
W20260926 07:24:06.100826 19564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:06.100841 19561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:06.100884 19555 server_base.cc:1034] running on GCE node
W20260926 07:24:06.100847 19562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:06.101866 19555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:06.103219 19555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:06.104445 19555 hybrid_clock.cc:648] HybridClock initialized: now 1790407446104420 us; error 44 us; skew 500 ppm
Sep 26 07:24:06 dist-test-slave-7zlw krb5kdc[19393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:06.107486 19555 init.cc:377] Logged in from keytab as kudu/127.28.235.252@KRBTEST.COM (short username kudu)
I20260926 07:24:06.108434 19555 webserver.cc:460] Webserver started at http://127.28.235.252:35421/ using document root <none> and password file <none>
I20260926 07:24:06.109057 19555 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:06.109166 19555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:06.109402 19555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:06.111465 19555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/data/instance:
uuid: "f81198900e914afca2c31c7f4f55606f"
format_stamp: "Formatted at 2026-09-26 07:24:06 on dist-test-slave-7zlw"
server_key: "efc0dda587e79d262507e65c79523f6f"
server_key_iv: "59311af09bfd7fb2c51c2debb5e486f9"
server_key_version: "encryptionkey@0"
I20260926 07:24:06.112058 19555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/wal/instance:
uuid: "f81198900e914afca2c31c7f4f55606f"
format_stamp: "Formatted at 2026-09-26 07:24:06 on dist-test-slave-7zlw"
server_key: "efc0dda587e79d262507e65c79523f6f"
server_key_iv: "59311af09bfd7fb2c51c2debb5e486f9"
server_key_version: "encryptionkey@0"
I20260926 07:24:06.116060 19555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 07:24:06.118845 19571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:06.120322 19555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:24:06.120505 19555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/wal
uuid: "f81198900e914afca2c31c7f4f55606f"
format_stamp: "Formatted at 2026-09-26 07:24:06 on dist-test-slave-7zlw"
server_key: "efc0dda587e79d262507e65c79523f6f"
server_key_iv: "59311af09bfd7fb2c51c2debb5e486f9"
server_key_version: "encryptionkey@0"
I20260926 07:24:06.120656 19555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:06.135152 19474 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } attempt: 1
I20260926 07:24:06.135442 19555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:06.136188 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.135805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50658)
0926 07:24:06.135969 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.136058 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 07:24:06.137100 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.136802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50666)
0926 07:24:06.136942 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.136996 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 07:24:06.137248 19474 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:44885: Network error: Client connection negotiation failed: client connection to 127.28.235.252:44885: connect: Connection refused (error 111)
I20260926 07:24:06.139274 19555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:06.139490 19555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:06.147234 19555 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.252:44885
I20260926 07:24:06.147856 19623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.252:44885 every 8 connection(s)
I20260926 07:24:06.148366 19555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
I20260926 07:24:06.151265 19624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:06.154976 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 19555
I20260926 07:24:06.155094 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407154183638-29615-0/minicluster-data/master-2/wal/instance
I20260926 07:24:06.155392 29615 external_mini_cluster.cc:1428] Setting key c5eaf78fadcdb70c0f2dcc7653781545
I20260926 07:24:06.155314 19624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:24:06 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
Sep 26 07:24:06 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
I20260926 07:24:06.170414 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.157596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.28.235.254:33437)
0926 07:24:06.157726 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.157733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.157998 (+ 265us) server_negotiation.cc:407] Connection header received
0926 07:24:06.158392 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.158395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.158450 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.158531 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.159616 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.160477 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.161950 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.162343 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.162456 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.165609 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.165616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.165623 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.165666 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.167654 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.168286 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.168291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.168293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.168388 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.168973 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.168977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.168978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.169315 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.169392 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.170073 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.170220 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:24:06.171249 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.156833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.28.235.252:44885)
0926 07:24:06.157273 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.157285 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.157318 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:24:06.158121 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.158140 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.158363 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.158553 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.159525 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.160768 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.161950 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.162499 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.162647 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.166254 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.166261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.166278 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.166309 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.168356 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.169000 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.169006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.169008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.169136 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.169455 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.169460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.169466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.169725 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.169938 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.170181 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.170374 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":309,"spinlock_wait_cycles":896,"thread_start_us":152,"threads_started":1}
I20260926 07:24:06.171248 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.157516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:33437 (local address 127.0.0.1:54370)
0926 07:24:06.157851 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.157899 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:06.158213 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:06.158603 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:06.158609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:06.159077 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:06.159450 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.159474 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.160597 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.160599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.161789 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.161800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.162493 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.162496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.162659 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.163466 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:06.163489 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:06.165416 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:06.167815 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.167826 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.167839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.168176 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.168514 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.168517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.168519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.168808 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.169426 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:06.169436 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:06.169737 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 07:24:06.169944 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 07:24:06.170312 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20260926 07:24:06.173265 19624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:24:06 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
Sep 26 07:24:06 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:24:06.183387 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.173786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:36735 (local address 127.0.0.1:51604)
0926 07:24:06.173948 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.173964 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:06.174037 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:06.174318 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:06.174321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:06.174515 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:06.174665 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.174673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.175659 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.175662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.176520 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.176531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.177154 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.177157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.177278 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.177768 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:06.177780 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:06.179299 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:06.181512 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.181515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.181517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.181778 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.182104 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.182107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.182109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.182221 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.182796 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:06.182800 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:06.182943 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 07:24:06.183068 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 07:24:06.183208 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 07:24:06.183344 19549 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } attempt: 3
I20260926 07:24:06.184096 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.173888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.28.235.253:36735)
0926 07:24:06.173999 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.174002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.174016 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:24:06.174146 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.174149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.174197 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.174281 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.174796 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.175544 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.176670 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.177012 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.177164 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.179456 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.179461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.179464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.179498 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.181374 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.181886 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.181890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.181893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.181983 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.182349 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.182353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.182354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.182667 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.182774 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.183192 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.183339 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:24:06.184743 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.173875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.28.235.254:33437)
0926 07:24:06.173982 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.173986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.174001 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:24:06.174151 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.174154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.174197 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.174281 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.174993 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.175710 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.176805 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.177181 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.177290 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.180033 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.180036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.180038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.180061 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.181909 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.182501 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.182505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.182506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.182599 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.183119 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.183123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.183125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.183400 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.183667 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.184414 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.184561 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:24:06.184904 19624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:24:06 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
Sep 26 07:24:06 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407446, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
Sep 26 07:24:06 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
I20260926 07:24:06.193537 19552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.183843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50682)
0926 07:24:06.183984 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.184002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:06.184089 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:06.184623 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:06.184626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:06.184813 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:06.184920 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.184931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.186220 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.186224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.187044 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.187058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.187580 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.187584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.187707 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.188260 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:06.188274 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:06.189688 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:06.191789 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.191795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.191800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.192058 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.192439 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.192443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.192445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.192548 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.192941 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:06.192945 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:06.193073 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 07:24:06.193203 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 07:24:06.193356 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":27}
I20260926 07:24:06.193537 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.183893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.28.235.252:44885)
0926 07:24:06.184325 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.184333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.184348 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:24:06.184385 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.184388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.184434 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.184586 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.185388 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.186094 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.187184 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.187448 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.187579 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.189836 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.189847 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.189851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.189873 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.191660 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.192215 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.192220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.192225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.192326 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.192639 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.192642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.192644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.192849 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.192937 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.193268 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.193377 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260926 07:24:06.196427 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.185742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50694)
0926 07:24:06.186141 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.186157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:06.186302 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:06.187202 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:06.187207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:06.187391 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:06.187484 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.187491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.188894 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.188897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.189689 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.189697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.190375 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.190379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.190455 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.190856 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:06.190875 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:06.192463 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:06.194770 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.194775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.194778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.195049 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.195362 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.195365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.195368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.195452 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.195910 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:06.195913 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:06.195999 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 07:24:06.196101 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:24:06.196245 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320}
I20260926 07:24:06.196647 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.185767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.28.235.252:44885)
0926 07:24:06.186900 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.186909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.186923 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:24:06.186958 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.186961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.187002 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.187106 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.187610 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.188784 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.189789 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.190158 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.190263 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.192619 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.192623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.192627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.192653 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.194643 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.195171 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.195174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.195176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.195262 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.195556 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.195563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.195564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.195834 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.195928 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.196203 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.196332 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":278,"threads_started":1}
I20260926 07:24:06.197491 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.186323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.28.235.253:36735)
0926 07:24:06.186428 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.186432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.186549 (+ 117us) server_negotiation.cc:407] Connection header received
0926 07:24:06.186737 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.186739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.186777 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.186873 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.187633 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.188456 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.189596 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.189916 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.190328 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.193507 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.193513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.193516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.193548 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.195351 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.195853 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.195857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.195858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.195945 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.196260 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.196264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.196266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.196489 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.197088 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.197233 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.197343 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260926 07:24:06.197999 19549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3: Bootstrap starting.
I20260926 07:24:06.201174 19549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:06.201397 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:24:06.202183 19549 log.cc:824] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:06.203003 19448 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54380
I20260926 07:24:06.204418 19474 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } attempt: 2
I20260926 07:24:06.205694 19624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f: Bootstrap starting.
I20260926 07:24:06.208549 19624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:06.209676 19624 log.cc:824] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:24:06 dist-test-slave-7zlw krb5kdc[19393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407445, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
I20260926 07:24:06.211920 19549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3: No bootstrap required, opened a new log
I20260926 07:24:06.212011 19624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f: No bootstrap required, opened a new log
I20260926 07:24:06.215088 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.205187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.28.235.252:44885)
0926 07:24:06.205306 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.205310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.205327 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:24:06.205368 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.205371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.205418 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.205502 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.206136 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.206965 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.208037 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.208402 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.208495 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.211217 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.211224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.211230 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.211260 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.213164 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.213719 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.213723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.213725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.213818 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.214121 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.214125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.214126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.214359 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.214461 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.214797 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.214924 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:24:06.215101 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.204873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:44885 (local address 127.0.0.1:50696)
0926 07:24:06.205116 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:06.205134 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:06.205228 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:06.205559 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:06.205563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:06.205812 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:06.205956 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.205965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.207099 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.207104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.207894 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:06.207907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.208539 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.208543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:06.208643 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.209345 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:06.209364 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:06.211044 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:06.213305 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.213309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.213311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.213596 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.213915 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:06.213919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:06.213921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:06.214019 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:06.214466 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:06.214470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:06.214603 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 07:24:06.214733 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 07:24:06.214893 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260926 07:24:06.216300 19624 raft_consensus.cc:348] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } }
I20260926 07:24:06.216544 19624 raft_consensus.cc:729] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81198900e914afca2c31c7f4f55606f, State: Initialized, Role: FOLLOWER
I20260926 07:24:06.216563 19549 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } }
I20260926 07:24:06.216799 19549 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb3a1fdf95248728691c39445c09ed3, State: Initialized, Role: FOLLOWER
I20260926 07:24:06.216907 19624 consensus_queue.cc:260] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } }
I20260926 07:24:06.217163 19549 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } }
I20260926 07:24:06.217725 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } } }
I20260926 07:24:06.217926 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:06.218366 19624 sys_catalog.cc:564] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:06.218509 19549 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:06.218740 19474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4: Bootstrap starting.
I20260926 07:24:06.221985 19474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:06.222929 19474 log.cc:824] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4: Log is configured to *not* fsync() on all Append() calls
W20260926 07:24:06.224843 19655 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:24:06.224997 19655 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:24:06.225991 19474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4: No bootstrap required, opened a new log
I20260926 07:24:06.222728 19555 master_runner.cc:186] Error getting master registration for 127.28.235.254:33437: OK, instance_id { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" instance_seqno: 1790407445671909 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:24:06.226435 19555 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:24:06.223184 19481 master_runner.cc:186] Error getting master registration for 127.28.235.254:33437: OK, instance_id { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" instance_seqno: 1790407445671909 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:24:06.227325 19481 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:24:06.229415 19474 raft_consensus.cc:348] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } }
I20260926 07:24:06.229712 19474 raft_consensus.cc:729] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda7cd6c48e641ffa3c8a72bf880aca4, State: Initialized, Role: FOLLOWER
I20260926 07:24:06.230077 19474 consensus_queue.cc:260] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } }
I20260926 07:24:06.231583 19474 sys_catalog.cc:564] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:06.231657 19448 raft_consensus.cc:482] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:24:06.231773 19448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [term 0 FOLLOWER]: Advancing to term 1
W20260926 07:24:06.232442 19654 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:24:06.232506 19654 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:24:06.233091 19632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } } }
I20260926 07:24:06.233281 19448 raft_consensus.cc:504] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } }
I20260926 07:24:06.233343 19632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:06.234344 19448 leader_election.cc:288] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [CANDIDATE]: Term 1 election: Requested vote from peers eeb3a1fdf95248728691c39445c09ed3 (127.28.235.253:36735), f81198900e914afca2c31c7f4f55606f (127.28.235.252:44885)
I20260926 07:24:06.235105 19598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f81198900e914afca2c31c7f4f55606f"
I20260926 07:24:06.235124 19523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eeb3a1fdf95248728691c39445c09ed3"
I20260926 07:24:06.235342 19598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:06.235370 19523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:06.236779 19598 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eda7cd6c48e641ffa3c8a72bf880aca4 in term 1.
I20260926 07:24:06.236886 19523 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eda7cd6c48e641ffa3c8a72bf880aca4 in term 1.
I20260926 07:24:06.236842 19658 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } } }
I20260926 07:24:06.236984 19658 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:06.237452 19423 leader_election.cc:302] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eda7cd6c48e641ffa3c8a72bf880aca4, eeb3a1fdf95248728691c39445c09ed3; no voters:
I20260926 07:24:06.239540 19658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:24:06.239784 19658 raft_consensus.cc:686] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [term 1 LEADER]: Becoming Leader. State: Replica: eda7cd6c48e641ffa3c8a72bf880aca4, State: Running, Role: LEADER
I20260926 07:24:06.240170 19658 consensus_queue.cc:237] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } }
W20260926 07:24:06.240939 19670 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:24:06.241081 19670 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:24:06.242314 19658 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eda7cd6c48e641ffa3c8a72bf880aca4. Latest consensus state: current_term: 1 leader_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } } }
I20260926 07:24:06.242439 19658 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:06.243870 19406 master_runner.cc:263] No leader master found from master eda7cd6c48e641ffa3c8a72bf880aca4
I20260926 07:24:06.243963 19406 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:24:06.245416 19681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:24:06.247725 19681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:24:06.266896 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.238786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51616 (local address 127.28.235.253:36735)
0926 07:24:06.238906 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.238911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.253500 (+ 14589us) server_negotiation.cc:407] Connection header received
0926 07:24:06.255849 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.255853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.255911 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.255970 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.258195 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.258987 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.260281 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.260672 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.260754 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.262822 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.262827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.262830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.262883 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.264776 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.265356 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.265360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.265362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.265459 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.265814 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.265818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.265819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.266054 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.266179 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.266574 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.266712 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:24:06.269819 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.237577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.28.235.252:44885)
0926 07:24:06.237711 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.237717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.251301 (+ 13584us) server_negotiation.cc:407] Connection header received
0926 07:24:06.251565 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.251570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.251629 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.251691 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.253603 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.254435 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.257097 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.257486 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.257586 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.266164 (+ 8578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.266169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.266172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.266207 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.267913 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.268410 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.268413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.268415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.268498 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.268821 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.268825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.268827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.269060 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.269188 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.269516 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.269653 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:24:06.270581 19598 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [term 1 FOLLOWER]: Refusing update from remote peer eda7cd6c48e641ffa3c8a72bf880aca4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:24:06.271109 19523 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [term 1 FOLLOWER]: Refusing update from remote peer eda7cd6c48e641ffa3c8a72bf880aca4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:24:06.271318 19679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:24:06.271792 19679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:24:06.273794 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:06.238464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.28.235.254:33437)
0926 07:24:06.238591 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.238598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.253137 (+ 14539us) server_negotiation.cc:407] Connection header received
0926 07:24:06.253264 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.253267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.253327 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.253438 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.262083 (+ 8645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.262914 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.267596 (+ 4682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:06.267915 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:06.268020 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:06.269860 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:06.269867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:06.269873 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:06.269906 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:06.271782 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.272320 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.272324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.272326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.272414 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:06.272733 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:06.272738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:06.272740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:06.272957 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:06.273125 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:06.273489 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 07:24:06.273618 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:24:06.276528 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eda7cd6c48e641ffa3c8a72bf880aca4. Latest consensus state: current_term: 1 leader_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } } }
I20260926 07:24:06.276710 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:06.280647 19632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eda7cd6c48e641ffa3c8a72bf880aca4. Latest consensus state: current_term: 1 leader_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } } }
I20260926 07:24:06.280812 19632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:06.281692 19658 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } } }
I20260926 07:24:06.281836 19658 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:06.281972 19682 mvcc.cc:205] Tried to move back new op lower bound from 7333508899907530752 to 7333508899803746304. Current Snapshot: MvccSnapshot[applied={T|T < 7333508899907530752}]
I20260926 07:24:06.283789 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } } }
I20260926 07:24:06.283929 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:06.287382 19681 catalog_manager.cc:1347] Generated new cluster ID: fcc55153a00441f0b2ca0cbbc48f9fde
I20260926 07:24:06.287464 19681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:24:06.291277 19679 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } } }
I20260926 07:24:06.291451 19679 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:06.292797 19632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda7cd6c48e641ffa3c8a72bf880aca4" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 33437 } } peers { permanent_uuid: "eeb3a1fdf95248728691c39445c09ed3" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 36735 } } peers { permanent_uuid: "f81198900e914afca2c31c7f4f55606f" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 44885 } } }
I20260926 07:24:06.292943 19632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:06.297127 19685 mvcc.cc:205] Tried to move back new op lower bound from 7333508899907530752 to 7333508899803746304. Current Snapshot: MvccSnapshot[applied={T|T < 7333508899907530752}]
I20260926 07:24:06.308079 19681 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:24:06.309531 19681 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:24:06.322790 19681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eda7cd6c48e641ffa3c8a72bf880aca4: Generated new TSK 0
I20260926 07:24:06.323868 19681 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:24:06.352814 19629 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.349068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50702 (local address 127.28.235.252:44885)
0926 07:24:06.349219 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.349225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.351506 (+ 2281us) server_negotiation.cc:407] Connection header received
0926 07:24:06.351737 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.351744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.351804 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.351866 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.352652 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50702: BlockingRecv error: recv got EOF from 127.0.0.1:50702 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 07:24:06.354382 19477 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.350082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.28.235.254:33437)
0926 07:24:06.350214 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.350219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.353300 (+ 3081us) server_negotiation.cc:407] Connection header received
0926 07:24:06.353501 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.353504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.353561 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.353622 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.354242 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54390: BlockingRecv error: recv got EOF from 127.0.0.1:54390 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 07:24:06.356165 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.350808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.28.235.253:36735)
0926 07:24:06.350933 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.350938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.354936 (+ 3998us) server_negotiation.cc:407] Connection header received
0926 07:24:06.355358 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.355362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.355419 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.355477 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.356034 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51620: BlockingRecv error: recv got EOF from 127.0.0.1:51620 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 07:24:06.361425 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.359560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.28.235.253:36735)
0926 07:24:06.359679 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.359684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.360240 (+ 556us) server_negotiation.cc:407] Connection header received
0926 07:24:06.360495 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.360500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.360591 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.360653 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.361305 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51628: BlockingRecv error: recv got EOF from 127.0.0.1:51628 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 07:24:06.367116 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.365340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.28.235.253:36735)
0926 07:24:06.365482 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.365487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.366041 (+ 554us) server_negotiation.cc:407] Connection header received
0926 07:24:06.366252 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.366256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.366312 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.366370 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.366914 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51638: BlockingRecv error: recv got EOF from 127.0.0.1:51638 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 07:24:06.373377 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.371595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.28.235.253:36735)
0926 07:24:06.371712 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.371717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.372283 (+ 566us) server_negotiation.cc:407] Connection header received
0926 07:24:06.372503 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.372507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.372567 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.372628 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.373250 (+ 622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51640: BlockingRecv error: recv got EOF from 127.0.0.1:51640 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 07:24:06.382238 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.380021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.28.235.253:36735)
0926 07:24:06.380191 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.380197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.380915 (+ 718us) server_negotiation.cc:407] Connection header received
0926 07:24:06.381201 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.381205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.381268 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.381330 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.382101 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51644: BlockingRecv error: recv got EOF from 127.0.0.1:51644 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260926 07:24:06.395713 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.393778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.28.235.253:36735)
0926 07:24:06.393913 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.393919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.394570 (+ 651us) server_negotiation.cc:407] Connection header received
0926 07:24:06.394813 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.394817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.394879 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.394943 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.395582 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51658: BlockingRecv error: recv got EOF from 127.0.0.1:51658 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 07:24:06.416654 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.414717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.28.235.253:36735)
0926 07:24:06.414850 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.414855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.415465 (+ 610us) server_negotiation.cc:407] Connection header received
0926 07:24:06.415795 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.415799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.415861 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.415921 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.416516 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51668: BlockingRecv error: recv got EOF from 127.0.0.1:51668 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 07:24:06.452718 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.451377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.28.235.253:36735)
0926 07:24:06.451503 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.451508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.451798 (+ 290us) server_negotiation.cc:407] Connection header received
0926 07:24:06.451971 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.451974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.452033 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.452126 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.452597 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51678: BlockingRecv error: recv got EOF from 127.0.0.1:51678 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 07:24:06.520663 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.518974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.28.235.253:36735)
0926 07:24:06.519180 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.519184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.519291 (+ 107us) server_negotiation.cc:407] Connection header received
0926 07:24:06.519774 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.519781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.519847 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.519931 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.520495 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51692: BlockingRecv error: recv got EOF from 127.0.0.1:51692 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260926 07:24:06.651955 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.650717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.28.235.253:36735)
0926 07:24:06.650890 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.650894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.650988 (+ 94us) server_negotiation.cc:407] Connection header received
0926 07:24:06.651165 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.651168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.651226 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.651321 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.651824 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51702: BlockingRecv error: recv got EOF from 127.0.0.1:51702 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 07:24:06.911316 19554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:06.910297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.28.235.253:36735)
0926 07:24:06.910441 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:06.910445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:06.910477 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:24:06.910624 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:06.910627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:06.910685 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:06.910768 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:06.911212 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51706: BlockingRecv error: recv got EOF from 127.0.0.1:51706 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:24:07.226722 19655 catalog_manager.cc:1259] Loaded cluster ID: fcc55153a00441f0b2ca0cbbc48f9fde
I20260926 07:24:07.226817 19655 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f: loading cluster ID for follower catalog manager: success
I20260926 07:24:07.228574 19655 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f: acquiring CA information for follower catalog manager: success
I20260926 07:24:07.229868 19655 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f81198900e914afca2c31c7f4f55606f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:24:07.233871 19654 catalog_manager.cc:1259] Loaded cluster ID: fcc55153a00441f0b2ca0cbbc48f9fde
I20260926 07:24:07.233935 19654 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3: loading cluster ID for follower catalog manager: success
I20260926 07:24:07.235483 19654 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3: acquiring CA information for follower catalog manager: success
I20260926 07:24:07.236725 19654 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eeb3a1fdf95248728691c39445c09ed3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:24:07.429620 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:07.425509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.28.235.253:36735)
0926 07:24:07.425767 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:07.425772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:07.425862 (+ 90us) server_negotiation.cc:407] Connection header received
0926 07:24:07.425987 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:07.425990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:07.426117 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:07.426229 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:24:07.426472 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:07.426987 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:07.427924 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:07.428177 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:07.428278 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:07.428580 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:24:07.429121 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:24:07.429315 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:07.429349 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 07:24:07.429408 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":68,"threads_started":1}
I20260926 07:24:07.436764 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:07.432875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.28.235.252:44885)
0926 07:24:07.433176 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:07.433179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:07.433193 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:24:07.433360 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:07.433362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:07.433479 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:07.433619 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:24:07.433841 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:07.434376 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:07.435180 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:07.435421 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:07.435501 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:07.435802 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:24:07.436294 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:24:07.436398 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:07.436500 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 07:24:07.436548 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":91,"threads_started":1}
I20260926 07:24:07.444129 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:07.439882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.28.235.254:33437)
0926 07:24:07.440141 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:07.440146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:07.440255 (+ 109us) server_negotiation.cc:407] Connection header received
0926 07:24:07.440431 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:07.440434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:07.440599 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:07.440723 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:24:07.441081 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:07.441594 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:07.442454 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:07.442690 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:07.442794 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:07.443203 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:24:07.443480 (+ 277us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:24:07.443589 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:07.443745 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 07:24:07.443810 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260926 07:24:07.444504 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:07.440339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.28.235.253:36735)
0926 07:24:07.440485 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:07.440489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:07.440581 (+ 92us) server_negotiation.cc:407] Connection header received
0926 07:24:07.440791 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:07.440794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:07.440851 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:07.440901 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:24:07.441289 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:07.442032 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:07.443141 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:07.443471 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:07.443545 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:07.443980 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:24:07.444089 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:24:07.444159 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:07.444319 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 07:24:07.444375 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:24:07.444556 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:07.439960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.28.235.252:44885)
0926 07:24:07.440088 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:07.440092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:07.440539 (+ 447us) server_negotiation.cc:407] Connection header received
0926 07:24:07.440588 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:07.440591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:07.440635 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:07.440709 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:24:07.440964 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:07.441821 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:07.443106 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:07.443449 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:07.443551 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:07.443845 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:24:07.443926 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:24:07.443997 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:07.444119 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 07:24:07.444179 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:24:07.447706 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 19406
I20260926 07:24:07.454468 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 19481
I20260926 07:24:07.460585 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 19555
2026-09-26T07:24:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3160 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:24:07 dist-test-slave-7zlw krb5kdc[19773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:24:07 dist-test-slave-7zlw krb5kdc[19773](info): set up 2 sockets
Sep 26 07:24:07 dist-test-slave-7zlw krb5kdc[19773](info): commencing operation
krb5kdc: starting...
W20260926 07:24:08.529572 29615 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/test-user.keytab.
Sep 26 07:24:08 dist-test-slave-7zlw krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:24:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:24:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
Entry for principal kudu/127.28.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab.
I20260926 07:24:08.664350 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:45469
--webserver_interface=127.28.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:45469,127.28.235.253:40081,127.28.235.252:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:08.777457 19786 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:24:08.777769 19786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:08.777793 19786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:08.777834 19786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:08.782112 19786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:08.782187 19786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:08.782207 19786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:08.782224 19786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:08.782240 19786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:08.787736 19786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:45469,127.28.235.253:40081,127.28.235.252:45877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.254.keytab
--principal=kudu/127.28.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.254:45469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:08.788827 19786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:08.789899 19786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:08.798782 19794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:08.798786 19792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:08.798830 19791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:08.799181 19786 server_base.cc:1034] running on GCE node
I20260926 07:24:08.799757 19786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:08.800915 19786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:08.802124 19786 hybrid_clock.cc:648] HybridClock initialized: now 1790407448802108 us; error 36 us; skew 500 ppm
Sep 26 07:24:08 dist-test-slave-7zlw krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:08.805199 19786 init.cc:377] Logged in from keytab as kudu/127.28.235.254@KRBTEST.COM (short username kudu)
I20260926 07:24:08.806083 19786 webserver.cc:460] Webserver started at http://127.28.235.254:46067/ using document root <none> and password file <none>
I20260926 07:24:08.806627 19786 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:08.806687 19786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:08.806855 19786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:08.808995 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/data/instance:
uuid: "4317378f94f7459facb7c53b7b873598"
format_stamp: "Formatted at 2026-09-26 07:24:08 on dist-test-slave-7zlw"
server_key: "ea86c4c98cd5bedc90a5cfa96bdfea83"
server_key_iv: "71bd8eaf64fad05516269b64d1e12a23"
server_key_version: "encryptionkey@0"
I20260926 07:24:08.809612 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/wal/instance:
uuid: "4317378f94f7459facb7c53b7b873598"
format_stamp: "Formatted at 2026-09-26 07:24:08 on dist-test-slave-7zlw"
server_key: "ea86c4c98cd5bedc90a5cfa96bdfea83"
server_key_iv: "71bd8eaf64fad05516269b64d1e12a23"
server_key_version: "encryptionkey@0"
I20260926 07:24:08.813510 19786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:24:08.815892 19801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:08.817354 19786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:24:08.817471 19786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/wal
uuid: "4317378f94f7459facb7c53b7b873598"
format_stamp: "Formatted at 2026-09-26 07:24:08 on dist-test-slave-7zlw"
server_key: "ea86c4c98cd5bedc90a5cfa96bdfea83"
server_key_iv: "71bd8eaf64fad05516269b64d1e12a23"
server_key_version: "encryptionkey@0"
I20260926 07:24:08.817589 19786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:08.837574 19786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:08.841128 19786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:08.841307 19786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:08.848414 19786 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.254:45469
I20260926 07:24:08.848738 19853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.254:45469 every 8 connection(s)
I20260926 07:24:08.849521 19786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/data/info.pb
I20260926 07:24:08.851269 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 19786
I20260926 07:24:08.851416 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-0/wal/instance
I20260926 07:24:08.851727 29615 external_mini_cluster.cc:1428] Setting key c0aceee3a6ff94f6ba8fe58341f5c0a9
I20260926 07:24:08.852593 19854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:08.856886 19854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.235.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.253@KRBTEST.COM" created.
Sep 26 07:24:08 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:24:08.873711 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:08.859297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37004 (local address 127.28.235.254:45469)
0926 07:24:08.859859 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:08.859879 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:08.859999 (+ 120us) server_negotiation.cc:407] Connection header received
0926 07:24:08.861201 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:08.861206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:08.861411 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:08.861579 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:08.862496 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:08.863607 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:08.864894 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:08.865565 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:08.865746 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:08.868605 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:08.868613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:08.868679 (+ 66us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:08.868700 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:08.870474 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:08.870994 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:08.871000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:08.871002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:08.871111 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:08.871423 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:08.871427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:08.871428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:08.871926 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:08.872065 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:08.872437 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 07:24:08.872820 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":436,"spinlock_wait_cycles":2944,"thread_start_us":144,"threads_started":1}
I20260926 07:24:08.873713 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:08.859296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:45469 (local address 127.0.0.1:37004)
0926 07:24:08.859799 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:08.859849 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:08.860733 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:08.861604 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:08.861609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:08.862070 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:08.862340 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:08.862354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:08.863752 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:08.863759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:08.864769 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:08.864778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:08.865731 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:08.865734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:08.865881 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:08.866500 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:08.866517 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:08.868387 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:08.870604 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:08.870612 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:08.870623 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:08.870873 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:08.871209 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:08.871212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:08.871213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:08.871326 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:08.872044 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:08.872051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:08.872202 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 07:24:08.872377 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 07:24:08.872711 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":54,"spinlock_wait_cycles":9600,"thread_start_us":152,"threads_started":1}
I20260926 07:24:08.876164 19854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:24:08.877100 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:08.876795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:40081 (local address 127.0.0.1:46950)
0926 07:24:08.876916 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:08.876985 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:40081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:24:08.877331 19804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.235.253:40081: connect: Connection refused (error 111)
I20260926 07:24:08.878226 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:08.877998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:40081 (local address 127.0.0.1:46964)
0926 07:24:08.878129 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:08.878177 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:40081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 07:24:08.878397 19854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.253:40081: Network error: Client connection negotiation failed: client connection to 127.28.235.253:40081: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab.
Entry for principal kudu/127.28.235.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab.
I20260926 07:24:08.881366 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.253:40081
--webserver_interface=127.28.235.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab
--principal=kudu/127.28.235.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:45469,127.28.235.253:40081,127.28.235.252:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:24:08.927685 19854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } attempt: 1
I20260926 07:24:08.928625 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:08.928280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:40081 (local address 127.0.0.1:46974)
0926 07:24:08.928476 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:08.928545 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:40081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 07:24:08.929497 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:08.929285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:40081 (local address 127.0.0.1:46988)
0926 07:24:08.929392 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:08.929441 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:40081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:24:08.929666 19854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.253:40081: Network error: Client connection negotiation failed: client connection to 127.28.235.253:40081: connect: Connection refused (error 111)
W20260926 07:24:08.996064 19861 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:24:08.996415 19861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:08.996479 19861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:08.996580 19861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:24:08.997798 19854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } attempt: 2
I20260926 07:24:08.998817 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:08.998298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:40081 (local address 127.0.0.1:46992)
0926 07:24:08.998549 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:08.998667 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:40081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 07:24:08.999763 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:08.999466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:40081 (local address 127.0.0.1:46996)
0926 07:24:08.999615 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:08.999681 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.253:40081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:24:08.999975 19854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.253:40081: Network error: Client connection negotiation failed: client connection to 127.28.235.253:40081: connect: Connection refused (error 111)
W20260926 07:24:09.002866 19861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:09.002960 19861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:09.003003 19861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:09.003042 19861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:09.003077 19861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:09.009920 19861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:45469,127.28.235.253:40081,127.28.235.252:45877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.253.keytab
--principal=kudu/127.28.235.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.253:40081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.235.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:09.011123 19861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:09.012207 19861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:09.021260 19866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:09.021288 19867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:09.021291 19861 server_base.cc:1034] running on GCE node
W20260926 07:24:09.021270 19869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:09.022184 19861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:09.023370 19861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:09.024564 19861 hybrid_clock.cc:648] HybridClock initialized: now 1790407449024542 us; error 38 us; skew 500 ppm
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407449, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:09.027663 19861 init.cc:377] Logged in from keytab as kudu/127.28.235.253@KRBTEST.COM (short username kudu)
I20260926 07:24:09.028671 19861 webserver.cc:460] Webserver started at http://127.28.235.253:41431/ using document root <none> and password file <none>
I20260926 07:24:09.029363 19861 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:09.029456 19861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:09.029703 19861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:09.031809 19861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/data/instance:
uuid: "c7c64512259a46e78e45ac4b8f28fd38"
format_stamp: "Formatted at 2026-09-26 07:24:09 on dist-test-slave-7zlw"
server_key: "ca10e890b313c631fd4227b1d1cf0b84"
server_key_iv: "1e31470a13df6a73d26450d856172c70"
server_key_version: "encryptionkey@0"
I20260926 07:24:09.032394 19861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/wal/instance:
uuid: "c7c64512259a46e78e45ac4b8f28fd38"
format_stamp: "Formatted at 2026-09-26 07:24:09 on dist-test-slave-7zlw"
server_key: "ca10e890b313c631fd4227b1d1cf0b84"
server_key_iv: "1e31470a13df6a73d26450d856172c70"
server_key_version: "encryptionkey@0"
I20260926 07:24:09.036286 19861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:24:09.038997 19876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:09.040422 19861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:24:09.040575 19861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/wal
uuid: "c7c64512259a46e78e45ac4b8f28fd38"
format_stamp: "Formatted at 2026-09-26 07:24:09 on dist-test-slave-7zlw"
server_key: "ca10e890b313c631fd4227b1d1cf0b84"
server_key_iv: "1e31470a13df6a73d26450d856172c70"
server_key_version: "encryptionkey@0"
I20260926 07:24:09.040735 19861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:09.059058 19861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:09.062516 19861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:09.062726 19861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:09.070284 19861 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.253:40081
I20260926 07:24:09.070614 19928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.253:40081 every 8 connection(s)
I20260926 07:24:09.071390 19861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/data/info.pb
I20260926 07:24:09.074208 19929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:09.078017 19929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:24:09.078552 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 19861
I20260926 07:24:09.078671 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-1/wal/instance
I20260926 07:24:09.078955 29615 external_mini_cluster.cc:1428] Setting key e03ac2ba9939ec1bd7680d9bfbe521ae
WARNING: no policy specified for kudu/127.28.235.252@KRBTEST.COM; defaulting to no policy
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407449, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:24:09.091195 19854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.235.252@KRBTEST.COM" created.
I20260926 07:24:09.095577 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.080097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.28.235.254:45469)
0926 07:24:09.080289 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.080296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.081304 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 07:24:09.082252 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.082255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.082320 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.082427 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.083565 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.084105 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.086056 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.086335 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.086440 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.089927 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.089930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.089932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.089961 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.092332 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.093230 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.093233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.093235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.093330 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.093796 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.093803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.093808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.094174 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.094300 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.095225 (+ 925us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.095354 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:24:09.096680 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.080543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:45469 (local address 127.0.0.1:37006)
0926 07:24:09.081117 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.081185 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:09.082037 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:09.082595 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:09.082602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:09.082953 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:09.083418 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.083431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.084251 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.084258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.085865 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.085878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.086521 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.086525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.086749 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.087524 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:09.087549 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:09.089683 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:09.092495 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.092506 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.092525 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.093117 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.093447 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.093450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.093452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.093640 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.094335 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:09.094344 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:09.094798 (+ 454us) client_negotiation.cc:769] Sending connection context
0926 07:24:09.095136 (+ 338us) client_negotiation.cc:240] Negotiation successful
0926 07:24:09.095529 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":10,"thread_start_us":148,"threads_started":1}
I20260926 07:24:09.098461 19929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
I20260926 07:24:09.104115 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.091688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:40081 (local address 127.0.0.1:47008)
0926 07:24:09.091921 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.091944 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:09.092034 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:09.093305 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:09.093308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:09.093483 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:09.093630 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.093646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.095507 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.095510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.096342 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.096356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.097192 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.097196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.097316 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.097877 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:09.097901 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:09.099729 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:09.102270 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.102280 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.102281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.102505 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.102879 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.102882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.102884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.102969 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.103503 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:09.103507 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:09.103627 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 07:24:09.103733 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:24:09.103891 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260926 07:24:09.104792 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.091832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.28.235.253:40081)
0926 07:24:09.092624 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.092642 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.092698 (+ 56us) server_negotiation.cc:407] Connection header received
0926 07:24:09.092776 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.092783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.093116 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.093312 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.094367 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.095371 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.096489 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.096989 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.098547 (+ 1558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.099906 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.099917 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.099951 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.099971 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.102130 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.102671 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.102677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.102679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.102770 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.103064 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.103067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.103069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.103369 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.103487 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.104452 (+ 965us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.104640 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":600,"thread_start_us":177,"threads_started":1}
I20260926 07:24:09.107120 19854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407449, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
I20260926 07:24:09.108022 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.107723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33090)
0926 07:24:09.107873 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.107940 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 07:24:09.109146 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.108741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33102)
0926 07:24:09.108967 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.109025 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260926 07:24:09.109287 19854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:45877: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
I20260926 07:24:09.111467 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.099017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:40081 (local address 127.0.0.1:47020)
0926 07:24:09.099135 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.099150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:09.099232 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:09.099828 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:09.099835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:09.100081 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:09.100249 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.100259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.102270 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.102274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.103199 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.103211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.103762 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.103766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.103881 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.104440 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:09.104451 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:09.107662 (+ 3211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:09.110018 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.110021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.110024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.110214 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.110492 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.110495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.110496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.110574 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.110949 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:09.110952 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:09.111036 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 07:24:09.111146 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:24:09.111294 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":3,"spinlock_wait_cycles":6016}
I20260926 07:24:09.111490 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.099228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.28.235.253:40081)
0926 07:24:09.099554 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.099563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.099576 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:24:09.099611 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.099614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.099677 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.099771 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.100375 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.101116 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.103328 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.103649 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.103762 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.107856 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.107860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.107863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.107891 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.109890 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.110308 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.110311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.110315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.110399 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.110663 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.110668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.110670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.110857 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.110962 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.111191 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.111351 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":267,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.235.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab.
Entry for principal kudu/127.28.235.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab.
I20260926 07:24:09.112262 19929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:24:09.112829 29615 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
/tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.252:45877
--webserver_interface=127.28.235.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab
--principal=kudu/127.28.235.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.235.212:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.235.254:45469,127.28.235.253:40081,127.28.235.252:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:24:09.112982 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.112710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33114)
0926 07:24:09.112822 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.112900 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:24:09.113212 19879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
I20260926 07:24:09.114040 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.113799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33122)
0926 07:24:09.113909 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.113966 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 07:24:09.114176 19929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:45877: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
I20260926 07:24:09.140476 19854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } attempt: 1
I20260926 07:24:09.141536 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.141239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33132)
0926 07:24:09.141397 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.141460 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 07:24:09.142231 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.142045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33140)
0926 07:24:09.142136 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.142183 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 07:24:09.142436 19854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:45877: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
I20260926 07:24:09.163499 19929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } attempt: 1
I20260926 07:24:09.164487 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.164127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33148)
0926 07:24:09.164311 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.164388 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 07:24:09.165402 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.165171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33158)
0926 07:24:09.165281 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.165336 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 07:24:09.165585 19929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:45877: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
I20260926 07:24:09.217629 19854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } attempt: 2
I20260926 07:24:09.218557 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.218247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33160)
0926 07:24:09.218413 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.218477 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 07:24:09.219358 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.219165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33170)
0926 07:24:09.219262 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.219305 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 07:24:09.219563 19854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:45877: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
W20260926 07:24:09.227465 19937 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:24:09.227775 19937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:09.227836 19937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:09.227926 19937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:09.232133 19937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:09.232199 19937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:09.232220 19937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:09.232236 19937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:09.232293 19937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:09.233757 19929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } attempt: 2
I20260926 07:24:09.234740 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.234400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33172)
0926 07:24:09.234570 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.234643 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 07:24:09.235574 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.235350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33176)
0926 07:24:09.235459 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.235510 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:24:09.235771 19929 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.235.252:45877: Network error: Client connection negotiation failed: client connection to 127.28.235.252:45877: connect: Connection refused (error 111)
I20260926 07:24:09.238571 19937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.235.212:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.235.254:45469,127.28.235.253:40081,127.28.235.252:45877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/krb5kdc/kudu_127.28.235.252.keytab
--principal=kudu/127.28.235.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.235.252:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.235.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:09.239951 19937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:09.241078 19937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:09.249780 19942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:09.249780 19945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:09.250011 19943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:09.250164 19937 server_base.cc:1034] running on GCE node
I20260926 07:24:09.250720 19937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:09.251855 19937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:09.253077 19937 hybrid_clock.cc:648] HybridClock initialized: now 1790407449253057 us; error 28 us; skew 500 ppm
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407449, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:09.255915 19937 init.cc:377] Logged in from keytab as kudu/127.28.235.252@KRBTEST.COM (short username kudu)
I20260926 07:24:09.256774 19937 webserver.cc:460] Webserver started at http://127.28.235.252:38527/ using document root <none> and password file <none>
I20260926 07:24:09.257385 19937 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:09.257448 19937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:09.257608 19937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:09.259703 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/data/instance:
uuid: "b2921e361d3a494e8d37704ad8276223"
format_stamp: "Formatted at 2026-09-26 07:24:09 on dist-test-slave-7zlw"
server_key: "05c3bcb5a41df93f022c0fed6758b21e"
server_key_iv: "bbc89da1b10138caad2eb988b1cc5d63"
server_key_version: "encryptionkey@0"
I20260926 07:24:09.260206 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/wal/instance:
uuid: "b2921e361d3a494e8d37704ad8276223"
format_stamp: "Formatted at 2026-09-26 07:24:09 on dist-test-slave-7zlw"
server_key: "05c3bcb5a41df93f022c0fed6758b21e"
server_key_iv: "bbc89da1b10138caad2eb988b1cc5d63"
server_key_version: "encryptionkey@0"
I20260926 07:24:09.263988 19937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 07:24:09.266547 19952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:09.267989 19937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:24:09.268229 19937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/data,/tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/wal
uuid: "b2921e361d3a494e8d37704ad8276223"
format_stamp: "Formatted at 2026-09-26 07:24:09 on dist-test-slave-7zlw"
server_key: "05c3bcb5a41df93f022c0fed6758b21e"
server_key_iv: "bbc89da1b10138caad2eb988b1cc5d63"
server_key_version: "encryptionkey@0"
I20260926 07:24:09.268365 19937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:09.282424 19937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:09.285804 19937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:09.286019 19937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:09.293321 19937 rpc_server.cc:307] RPC server started. Bound to: 127.28.235.252:45877
I20260926 07:24:09.293655 20004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.235.252:45877 every 8 connection(s)
I20260926 07:24:09.294409 19937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/data/info.pb
I20260926 07:24:09.297236 20005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:09.299551 29615 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu as pid 19937
I20260926 07:24:09.299708 29615 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZ0ZD4L/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407154183638-29615-0/minicluster-data/master-2/wal/instance
I20260926 07:24:09.299988 29615 external_mini_cluster.cc:1428] Setting key 2fe9969f8e37d315280625c74d729834
I20260926 07:24:09.301608 20005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407449, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
I20260926 07:24:09.314708 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.303495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37012 (local address 127.28.235.254:45469)
0926 07:24:09.303630 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.303634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.304016 (+ 382us) server_negotiation.cc:407] Connection header received
0926 07:24:09.304133 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.304145 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.304200 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.304296 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.305344 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.305850 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.306906 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.307115 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.307258 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.309945 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.309951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.309955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.309989 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.311729 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.312443 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.312447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.312449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.312550 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.313080 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.313083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.313087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.313413 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.313488 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.314350 (+ 862us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.314504 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:24:09.315977 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.301419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.28.235.252:45877)
0926 07:24:09.301916 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.301931 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.301966 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:24:09.302681 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.302699 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.303096 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.303424 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.304417 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.305729 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.306635 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.307108 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.307257 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.310368 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.310376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.310392 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.310422 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.312371 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.312870 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.312877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.312879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.312962 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.313363 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.313366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.313367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.314014 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.314301 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.314589 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.314761 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":352,"thread_start_us":182,"threads_started":1}
I20260926 07:24:09.315977 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.303457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.254:45469 (local address 127.0.0.1:37012)
0926 07:24:09.303851 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.303913 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:09.304037 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:09.304429 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:09.304435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:09.304894 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:09.305191 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.305207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.305970 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.305972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.306783 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.306791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.307308 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.307311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.307408 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.308111 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:09.308127 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:09.309703 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:09.311850 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.311858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.311869 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.312302 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.312678 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.312681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.312683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.312934 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.313514 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:09.313532 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:09.314014 (+ 482us) client_negotiation.cc:769] Sending connection context
0926 07:24:09.314274 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 07:24:09.314708 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":94,"spinlock_wait_cycles":1280,"thread_start_us":131,"threads_started":1}
I20260926 07:24:09.317445 20005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:24:09.318725 19854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } attempt: 3
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407449, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407448, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.254@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.254@KRBTEST.COM
I20260926 07:24:09.326951 19929 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } attempt: 3
I20260926 07:24:09.328745 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.318151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.28.235.253:40081)
0926 07:24:09.318281 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.318287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.318310 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:24:09.318435 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.318439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.318499 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.318588 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.319301 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.320193 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.321461 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.321772 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.321893 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.324725 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.324732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.324739 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.324780 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.326677 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.327284 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.327292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.327297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.327395 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.327712 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.327716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.327717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.327976 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.328070 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.328425 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.328573 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:24:09.329270 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.318023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.253:40081 (local address 127.0.0.1:47036)
0926 07:24:09.318238 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.318256 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:09.318327 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:09.318723 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:09.318725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:09.318963 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:09.319121 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.319130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.320373 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.320378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.321329 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.321340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.321891 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.321895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.321993 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.322630 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:09.322645 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:09.324565 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:09.326822 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.326826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.326829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.327181 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.327496 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.327499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.327504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.327594 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.328112 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:09.328115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:09.328238 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 07:24:09.328891 (+ 653us) client_negotiation.cc:240] Negotiation successful
0926 07:24:09.329073 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 07:24:09.329636 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.319107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33208)
0926 07:24:09.319221 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.319237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:09.319317 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:09.319645 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:09.319648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:09.319823 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:09.319930 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.319936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.320701 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.320705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.321384 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.321412 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.322043 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.322046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.322153 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.322819 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:09.322836 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:09.324565 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:09.327423 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.327429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.327433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.327753 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.328049 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.328053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.328055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.328146 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.328572 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:09.328579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:09.328698 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 07:24:09.328970 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 07:24:09.329170 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":20}
I20260926 07:24:09.330024 20005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:24:09.330852 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.319331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.28.235.252:45877)
0926 07:24:09.319454 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.319458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.319469 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:24:09.319506 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.319508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.319563 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.319642 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.320040 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.320595 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.321529 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.321895 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.322049 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.325137 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.325141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.325143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.325177 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.327313 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.327857 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.327859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.327861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.327932 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.328261 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.328264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.328265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.328456 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.328580 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.330471 (+ 1891us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.330673 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:24:09.330963 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.319293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.28.235.254:45469)
0926 07:24:09.319397 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.319402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.319570 (+ 168us) server_negotiation.cc:407] Connection header received
0926 07:24:09.319720 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.319724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.319769 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.319821 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.320800 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.321633 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.322872 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.323130 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.323257 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.326426 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.326429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.326432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.326460 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.328495 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.329133 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.329137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.329139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.329222 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.329569 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.329572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.329573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.329789 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.330569 (+ 780us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.330664 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.330788 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":768}
I20260926 07:24:09.335291 19854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598: Bootstrap starting.
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407449, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.253@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407449, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.235.252@KRBTEST.COM for kudu/127.28.235.252@KRBTEST.COM
Sep 26 07:24:09 dist-test-slave-7zlw krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.235.253@KRBTEST.COM
I20260926 07:24:09.339761 19854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:09.340711 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.327433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33224)
0926 07:24:09.327672 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.327690 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:09.327773 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:09.329622 (+ 1849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:09.329625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:09.329829 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:09.329939 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.329946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.332115 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.332119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.332994 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.333004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.333971 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.333975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.334082 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.334649 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:09.334667 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:09.336226 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:09.338603 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.338607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.338610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.338892 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.339275 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.339279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.339281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.339381 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.339925 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:09.339930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:09.340042 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 07:24:09.340137 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 07:24:09.340516 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 07:24:09.340736 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.330496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.235.252:45877 (local address 127.0.0.1:33232)
0926 07:24:09.331011 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:09.331026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:09.331103 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:09.331560 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:09.331562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:09.331709 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:09.331824 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.331833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.332850 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.332853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.333617 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:09.333627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.334130 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.334133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:09.334236 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.335602 (+ 1366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:09.335616 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:09.336978 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:09.339157 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.339160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.339162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.339377 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.339647 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:09.339650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:09.339652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:09.339733 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:09.340178 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:09.340181 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:09.340274 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 07:24:09.340372 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:24:09.340503 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":96,"threads_started":1}
I20260926 07:24:09.340785 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.328022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.28.235.252:45877)
0926 07:24:09.329340 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.329352 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.329370 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:24:09.329417 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.329422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.329473 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.330078 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.330198 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.331959 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.333468 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.333837 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.334336 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.336378 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.336383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.336386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.336413 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.338445 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.339042 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.339047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.339049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.339141 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.339514 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.339532 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.339537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.339795 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.340363 (+ 568us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.340459 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.340601 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":146,"threads_started":1}
I20260926 07:24:09.341321 19854 log.cc:824] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:09.341610 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.330813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.28.235.252:45877)
0926 07:24:09.331353 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.331360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.331372 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:24:09.331403 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.331405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.331442 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.331921 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.332059 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.332739 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.333739 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.334023 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.334421 (+ 398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.337170 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.337175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.337177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.337205 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.339039 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.339478 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.339481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.339482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.339556 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.339836 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.339842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.339846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.340061 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.341279 (+ 1218us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.341353 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.341457 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":78,"threads_started":1}
I20260926 07:24:09.344242 20005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223: Bootstrap starting.
I20260926 07:24:09.344419 19929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38: Bootstrap starting.
I20260926 07:24:09.345358 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.332223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.28.235.253:40081)
0926 07:24:09.332328 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.332332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.332619 (+ 287us) server_negotiation.cc:407] Connection header received
0926 07:24:09.332657 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.332660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.332703 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.332769 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.333695 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.334507 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.335549 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.335945 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.336066 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.339605 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.339609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.339612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.339637 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.342733 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.343310 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.343317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.343322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.343412 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.343744 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.343747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.343749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.343966 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.344154 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.345074 (+ 920us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.345197 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:24:09.347004 19854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598: No bootstrap required, opened a new log
I20260926 07:24:09.347185 19929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:09.347436 20005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:09.348140 19929 log.cc:824] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:09.348188 20005 log.cc:824] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:09.349104 29615 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:24:09.350482 19828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4317378f94f7459facb7c53b7b873598"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37020
I20260926 07:24:09.350732 20005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223: No bootstrap required, opened a new log
I20260926 07:24:09.350894 19929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38: No bootstrap required, opened a new log
I20260926 07:24:09.351379 19854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } }
I20260926 07:24:09.351609 19854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4317378f94f7459facb7c53b7b873598, State: Initialized, Role: FOLLOWER
I20260926 07:24:09.351945 19854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } }
I20260926 07:24:09.352617 20016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } } }
I20260926 07:24:09.352954 20016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:09.353513 19854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:09.353621 19828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:24:09.353747 19828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:09.354693 20005 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } }
I20260926 07:24:09.354944 20005 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2921e361d3a494e8d37704ad8276223, State: Initialized, Role: FOLLOWER
I20260926 07:24:09.355185 19828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } }
I20260926 07:24:09.355300 20005 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } }
I20260926 07:24:09.355453 19929 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } }
I20260926 07:24:09.355666 19929 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c64512259a46e78e45ac4b8f28fd38, State: Initialized, Role: FOLLOWER
I20260926 07:24:09.356240 19828 leader_election.cc:288] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [CANDIDATE]: Term 1 election: Requested vote from peers c7c64512259a46e78e45ac4b8f28fd38 (127.28.235.253:40081), b2921e361d3a494e8d37704ad8276223 (127.28.235.252:45877)
I20260926 07:24:09.357112 20005 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:09.357457 19979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4317378f94f7459facb7c53b7b873598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b2921e361d3a494e8d37704ad8276223"
I20260926 07:24:09.357683 19979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:09.359012 19979 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4317378f94f7459facb7c53b7b873598 in term 1.
I20260926 07:24:09.355962 19929 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } }
I20260926 07:24:09.359570 19804 leader_election.cc:302] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4317378f94f7459facb7c53b7b873598, b2921e361d3a494e8d37704ad8276223; no voters:
W20260926 07:24:09.360090 20028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:24:09.360183 20028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:24:09.360474 20027 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } } }
I20260926 07:24:09.360589 20027 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:09.360829 19929 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:09.360860 19903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4317378f94f7459facb7c53b7b873598" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c7c64512259a46e78e45ac4b8f28fd38"
I20260926 07:24:09.361095 19903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:09.362764 19903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4317378f94f7459facb7c53b7b873598 in term 1.
W20260926 07:24:09.364039 20045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:24:09.364106 20045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:24:09.363622 19786 master_runner.cc:186] Error getting master registration for 127.28.235.253:40081: OK, instance_id { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" instance_seqno: 1790407449067961 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:24:09.367517 19786 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:24:09.368747 20057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:24:09.368821 20057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:24:09.371701 20016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:24:09.371980 20016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [term 1 LEADER]: Becoming Leader. State: Replica: 4317378f94f7459facb7c53b7b873598, State: Running, Role: LEADER
I20260926 07:24:09.372323 19937 master_runner.cc:263] No leader master found from master b2921e361d3a494e8d37704ad8276223
I20260926 07:24:09.372390 19937 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:24:09.372344 20016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } }
I20260926 07:24:09.373338 19861 master_runner.cc:263] No leader master found from master c7c64512259a46e78e45ac4b8f28fd38
I20260926 07:24:09.373407 19861 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:24:09.374771 20062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4317378f94f7459facb7c53b7b873598. Latest consensus state: current_term: 1 leader_uuid: "4317378f94f7459facb7c53b7b873598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } } }
I20260926 07:24:09.374918 20062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:09.365218 20034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } } }
I20260926 07:24:09.375170 20034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:09.381309 20064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:24:09.382272 20064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:24:09.388182 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.364191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47066 (local address 127.28.235.253:40081)
0926 07:24:09.364308 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.364314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.377398 (+ 13084us) server_negotiation.cc:407] Connection header received
0926 07:24:09.377641 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.377645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.377699 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.377756 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.378898 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.379778 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.381207 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.381593 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.381678 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.384097 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.384102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.384105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.384134 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.386144 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.386716 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.386720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.386722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.386821 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.387144 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.387147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.387149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.387369 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.387482 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.387874 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.388003 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:24:09.396593 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.364893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.28.235.252:45877)
0926 07:24:09.365002 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.365006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.381515 (+ 16509us) server_negotiation.cc:407] Connection header received
0926 07:24:09.381813 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.381817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.381878 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.381940 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.383951 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.384893 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.389923 (+ 5030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.390340 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.390427 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.392351 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.392355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.392359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.392391 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.394431 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.395001 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.395005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.395007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.395119 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.395464 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.395467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.395470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.395726 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.395846 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.396248 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.396395 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:24:09.397233 19979 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [term 1 FOLLOWER]: Refusing update from remote peer 4317378f94f7459facb7c53b7b873598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:24:09.398003 20062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:24:09.399439 19903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [term 1 FOLLOWER]: Refusing update from remote peer 4317378f94f7459facb7c53b7b873598: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:24:09.402472 20027 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4317378f94f7459facb7c53b7b873598. Latest consensus state: current_term: 1 leader_uuid: "4317378f94f7459facb7c53b7b873598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } } }
I20260926 07:24:09.402621 20027 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:09.403047 20016 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:24:09.407460 20034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4317378f94f7459facb7c53b7b873598. Latest consensus state: current_term: 1 leader_uuid: "4317378f94f7459facb7c53b7b873598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } } }
I20260926 07:24:09.407600 20034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:09.410348 20016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4317378f94f7459facb7c53b7b873598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } } }
I20260926 07:24:09.410486 20016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:09.411887 20066 mvcc.cc:205] Tried to move back new op lower bound from 7333508912721563648 to 7333508912633282560. Current Snapshot: MvccSnapshot[applied={T|T < 7333508912721563648}]
I20260926 07:24:09.412148 20027 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4317378f94f7459facb7c53b7b873598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } } }
I20260926 07:24:09.412266 20027 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:09.414927 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:09.370322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.28.235.254:45469)
0926 07:24:09.370452 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.370457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.381303 (+ 10846us) server_negotiation.cc:407] Connection header received
0926 07:24:09.383062 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.383068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.383138 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.383204 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.385405 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.386292 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.388700 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:09.389133 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:09.389221 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:09.407032 (+ 17811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:09.407037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:09.407041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:09.407079 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:09.409038 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.413113 (+ 4075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.413120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.413124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.413226 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:09.413530 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:09.413533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:09.413535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:09.413772 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:09.413894 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:09.414635 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 07:24:09.414765 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:24:09.417798 20062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4317378f94f7459facb7c53b7b873598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } } }
I20260926 07:24:09.417937 20062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:09.419596 20034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4317378f94f7459facb7c53b7b873598" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4317378f94f7459facb7c53b7b873598" member_type: VOTER last_known_addr { host: "127.28.235.254" port: 45469 } } peers { permanent_uuid: "c7c64512259a46e78e45ac4b8f28fd38" member_type: VOTER last_known_addr { host: "127.28.235.253" port: 40081 } } peers { permanent_uuid: "b2921e361d3a494e8d37704ad8276223" member_type: VOTER last_known_addr { host: "127.28.235.252" port: 45877 } } }
I20260926 07:24:09.419735 20034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:24:09.419862 20068 mvcc.cc:205] Tried to move back new op lower bound from 7333508912721563648 to 7333508912633282560. Current Snapshot: MvccSnapshot[applied={T|T < 7333508912721563648}]
I20260926 07:24:09.426532 20064 catalog_manager.cc:1347] Generated new cluster ID: 7d8f875089cc4feda88ce0dd42a186f5
I20260926 07:24:09.426625 20064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:24:09.456362 20064 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:24:09.457842 20064 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:24:09.479372 20064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4317378f94f7459facb7c53b7b873598: Generated new TSK 0
I20260926 07:24:09.480356 20064 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:24:09.515959 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.512838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.28.235.253:40081)
0926 07:24:09.512974 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.512979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.514180 (+ 1201us) server_negotiation.cc:407] Connection header received
0926 07:24:09.515071 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.515075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.515133 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.515188 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.515819 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47078: BlockingRecv error: recv got EOF from 127.0.0.1:47078 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 07:24:09.517683 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.513630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.28.235.252:45877)
0926 07:24:09.513747 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.513752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.516364 (+ 2612us) server_negotiation.cc:407] Connection header received
0926 07:24:09.516691 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.516694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.516748 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.516799 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.517564 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33242: BlockingRecv error: recv got EOF from 127.0.0.1:33242 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 07:24:09.518740 19858 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.514476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.28.235.254:45469)
0926 07:24:09.514589 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.514594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.517084 (+ 2490us) server_negotiation.cc:407] Connection header received
0926 07:24:09.518060 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.518064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.518114 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.518164 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.518625 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37030: BlockingRecv error: recv got EOF from 127.0.0.1:37030 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 07:24:09.523391 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.521199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.28.235.253:40081)
0926 07:24:09.521299 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.521303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.522388 (+ 1085us) server_negotiation.cc:407] Connection header received
0926 07:24:09.522597 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.522600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.522672 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.522722 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.523285 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47092: BlockingRecv error: recv got EOF from 127.0.0.1:47092 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 07:24:09.528391 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.526896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.28.235.253:40081)
0926 07:24:09.527005 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.527009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.527498 (+ 489us) server_negotiation.cc:407] Connection header received
0926 07:24:09.527689 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.527692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.527742 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.527792 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.528290 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47096: BlockingRecv error: recv got EOF from 127.0.0.1:47096 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 07:24:09.534168 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.532507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.28.235.253:40081)
0926 07:24:09.532618 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.532622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.533201 (+ 579us) server_negotiation.cc:407] Connection header received
0926 07:24:09.533404 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.533408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.533463 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.533515 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.534052 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47102: BlockingRecv error: recv got EOF from 127.0.0.1:47102 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 07:24:09.542443 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.540585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.28.235.253:40081)
0926 07:24:09.540731 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.540736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.541430 (+ 694us) server_negotiation.cc:407] Connection header received
0926 07:24:09.541660 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.541664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.541719 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.541780 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.542328 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47108: BlockingRecv error: recv got EOF from 127.0.0.1:47108 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 07:24:09.554943 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.553011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.28.235.253:40081)
0926 07:24:09.553166 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.553173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.553797 (+ 624us) server_negotiation.cc:407] Connection header received
0926 07:24:09.554108 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.554112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.554173 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.554232 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.554818 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47116: BlockingRecv error: recv got EOF from 127.0.0.1:47116 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 07:24:09.575423 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.573556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.28.235.253:40081)
0926 07:24:09.573681 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.573686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.574311 (+ 625us) server_negotiation.cc:407] Connection header received
0926 07:24:09.574512 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.574516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.574570 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.574624 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.575292 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47124: BlockingRecv error: recv got EOF from 127.0.0.1:47124 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 07:24:09.610912 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.609609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.28.235.253:40081)
0926 07:24:09.609771 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.609777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.609971 (+ 194us) server_negotiation.cc:407] Connection header received
0926 07:24:09.610173 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.610176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.610226 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.610327 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.610805 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47126: BlockingRecv error: recv got EOF from 127.0.0.1:47126 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 07:24:09.678567 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.677262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.28.235.253:40081)
0926 07:24:09.677443 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.677448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.677592 (+ 144us) server_negotiation.cc:407] Connection header received
0926 07:24:09.677777 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.677780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.677836 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.677927 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.678446 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47134: BlockingRecv error: recv got EOF from 127.0.0.1:47134 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 07:24:09.810197 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:09.808800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.28.235.253:40081)
0926 07:24:09.808990 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:09.808996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:09.809147 (+ 151us) server_negotiation.cc:407] Connection header received
0926 07:24:09.809385 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:09.809388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:09.809445 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:09.809542 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:09.810040 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47142: BlockingRecv error: recv got EOF from 127.0.0.1:47142 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260926 07:24:10.069828 19935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:24:10.068486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.28.235.253:40081)
0926 07:24:10.068634 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:10.068638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:10.068654 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:24:10.068867 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:10.068870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:10.068922 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:10.069002 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:10.069636 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47144: BlockingRecv error: recv got EOF from 127.0.0.1:47144 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:24:10.365711 20045 catalog_manager.cc:1259] Loaded cluster ID: 7d8f875089cc4feda88ce0dd42a186f5
I20260926 07:24:10.365785 20045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223: loading cluster ID for follower catalog manager: success
I20260926 07:24:10.367384 20045 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223: acquiring CA information for follower catalog manager: success
I20260926 07:24:10.368491 20045 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2921e361d3a494e8d37704ad8276223: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:24:10.369855 20057 catalog_manager.cc:1259] Loaded cluster ID: 7d8f875089cc4feda88ce0dd42a186f5
I20260926 07:24:10.369915 20057 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38: loading cluster ID for follower catalog manager: success
I20260926 07:24:10.371541 20057 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38: acquiring CA information for follower catalog manager: success
I20260926 07:24:10.372648 20057 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7c64512259a46e78e45ac4b8f28fd38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:24:10.588188 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:10.583925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.28.235.253:40081)
0926 07:24:10.584198 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:10.584204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:10.584317 (+ 113us) server_negotiation.cc:407] Connection header received
0926 07:24:10.584511 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:10.584517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:10.584658 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:10.584786 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:24:10.585075 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:10.585639 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:10.586553 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:10.586799 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:10.586946 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:10.587291 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:24:10.587747 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:24:10.587886 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:10.587953 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:24:10.588014 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":70,"threads_started":1}
I20260926 07:24:10.595616 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:10.591647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.28.235.252:45877)
0926 07:24:10.591917 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:10.591924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:10.591941 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:24:10.592037 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:10.592040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:10.592161 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:10.592298 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:24:10.592531 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:10.593127 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:10.593979 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:10.594216 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:10.594324 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:10.594656 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:24:10.595216 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:24:10.595349 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:10.595390 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 07:24:10.595454 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260926 07:24:10.603529 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:10.599149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.28.235.252:45877)
0926 07:24:10.599261 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:10.599265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:10.599414 (+ 149us) server_negotiation.cc:407] Connection header received
0926 07:24:10.599479 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:10.599484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:10.599531 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:10.599591 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:24:10.599959 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:10.600475 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:10.601899 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:10.602225 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:10.602311 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:10.603026 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:24:10.603109 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:24:10.603198 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:10.603323 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:24:10.603379 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:24:10.603540 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:10.598773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.28.235.253:40081)
0926 07:24:10.598903 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:10.598906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:10.599378 (+ 472us) server_negotiation.cc:407] Connection header received
0926 07:24:10.599541 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:10.599544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:10.599604 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:10.599655 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:24:10.600030 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:10.600803 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:10.601849 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:10.602180 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:10.602296 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:10.603009 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:24:10.603121 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:24:10.603207 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:10.603328 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 07:24:10.603387 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:24:10.603821 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:10.598677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.28.235.254:45469)
0926 07:24:10.599074 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:10.599080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:10.599098 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:24:10.599215 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:10.599218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:10.599398 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:10.599525 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:24:10.599786 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:10.600622 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:10.601901 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:10.602140 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:10.602261 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:10.602597 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:24:10.602891 (+ 294us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:24:10.603008 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:10.603606 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 07:24:10.603673 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":75,"threads_started":1}
I20260926 07:24:10.607530 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 19786
I20260926 07:24:10.614790 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 19861
I20260926 07:24:10.621207 29615 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZ0ZD4L/build/debug/bin/kudu with pid 19937
2026-09-26T07:24:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3160 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6320 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (296428 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST